[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:45:16 dist-test-slave-rgb1 krb5kdc[16786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:45:16 dist-test-slave-rgb1 krb5kdc[16786](info): set up 2 sockets
Aug 07 22:45:16 dist-test-slave-rgb1 krb5kdc[16786](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260807 22:45:17.589857 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:45:17 dist-test-slave-rgb1 krb5kdc[16786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:45:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:45:17.737605 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:44795
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:44795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:17.824713 16799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:17.824939 16799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:17.824963 16799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:17.826138 16799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:17.826188 16799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:17.826203 16799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:17.826216 16799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:17.826228 16799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:17.827775 16799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:44795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:44795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:17.827998 16799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:17.828199 16799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:17.835858 16807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:17.835840 16805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:17.835840 16804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:17.835892 16799 server_base.cc:1034] running on GCE node
I20260807 22:45:17.836392 16799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:17.836789 16799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:17.837917 16799 hybrid_clock.cc:648] HybridClock initialized: now 1786142717837888 us; error 49 us; skew 500 ppm
Aug 07 22:45:17 dist-test-slave-rgb1 krb5kdc[16786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142717, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:17.841116 16799 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:45:17.841598 16799 webserver.cc:460] Webserver started at http://127.16.98.190:45185/ using document root <none> and password file <none>
I20260807 22:45:17.841887 16799 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:17.841958 16799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:17.842128 16799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:17.843392 16799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "ced65e913e2443db827ae18c099bb79b"
format_stamp: "Formatted at 2026-08-07 22:45:17 on dist-test-slave-rgb1"
server_key: "cd51d84054e9de07bfb23ab0460d185b"
server_key_iv: "943f991cfd107b05fd71fc36be6687f3"
server_key_version: "encryptionkey@0"
I20260807 22:45:17.843778 16799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "ced65e913e2443db827ae18c099bb79b"
format_stamp: "Formatted at 2026-08-07 22:45:17 on dist-test-slave-rgb1"
server_key: "cd51d84054e9de07bfb23ab0460d185b"
server_key_iv: "943f991cfd107b05fd71fc36be6687f3"
server_key_version: "encryptionkey@0"
I20260807 22:45:17.845422 16799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:45:17.846480 16814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:17.846764 16799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260807 22:45:17.846941 16799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "ced65e913e2443db827ae18c099bb79b"
format_stamp: "Formatted at 2026-08-07 22:45:17 on dist-test-slave-rgb1"
server_key: "cd51d84054e9de07bfb23ab0460d185b"
server_key_iv: "943f991cfd107b05fd71fc36be6687f3"
server_key_version: "encryptionkey@0"
I20260807 22:45:17.847038 16799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:17.855856 16799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:17.858649 16799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:17.858852 16799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:17.864925 16799 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:44795
I20260807 22:45:17.864982 16866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:44795 every 8 connection(s)
I20260807 22:45:17.865471 16799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:17.866590 16867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:17.869225 16867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b: Bootstrap starting.
I20260807 22:45:17.869907 16867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:17.870220 16867 log.cc:824] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:17.871197 16867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b: No bootstrap required, opened a new log
I20260807 22:45:17.873000 16867 raft_consensus.cc:348] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced65e913e2443db827ae18c099bb79b" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44795 } }
I20260807 22:45:17.873158 16867 raft_consensus.cc:374] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:17.873183 16867 raft_consensus.cc:729] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced65e913e2443db827ae18c099bb79b, State: Initialized, Role: FOLLOWER
I20260807 22:45:17.873340 16867 consensus_queue.cc:260] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced65e913e2443db827ae18c099bb79b" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44795 } }
I20260807 22:45:17.873485 16867 raft_consensus.cc:388] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:17.873513 16867 raft_consensus.cc:482] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:17.873548 16867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:17.873960 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 16799
I20260807 22:45:17.874042 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:17.874121 16867 raft_consensus.cc:504] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced65e913e2443db827ae18c099bb79b" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44795 } }
I20260807 22:45:17.874202 16867 leader_election.cc:302] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ced65e913e2443db827ae18c099bb79b; no voters:
I20260807 22:45:17.874396 16867 leader_election.cc:288] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:17.874429 16870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:17.874567 16870 raft_consensus.cc:686] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [term 1 LEADER]: Becoming Leader. State: Replica: ced65e913e2443db827ae18c099bb79b, State: Running, Role: LEADER
I20260807 22:45:17.874642 16870 consensus_queue.cc:237] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced65e913e2443db827ae18c099bb79b" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44795 } }
I20260807 22:45:17.874684 16867 sys_catalog.cc:564] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:17.874797 16778 external_mini_cluster.cc:1428] Setting key e77bf26a7ec3f42d9598109a6c273271
I20260807 22:45:17.875464 16871 sys_catalog.cc:455] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ced65e913e2443db827ae18c099bb79b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced65e913e2443db827ae18c099bb79b" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44795 } } }
I20260807 22:45:17.875576 16871 sys_catalog.cc:458] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:17.876267 16879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:17.876444 16872 sys_catalog.cc:455] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ced65e913e2443db827ae18c099bb79b. Latest consensus state: current_term: 1 leader_uuid: "ced65e913e2443db827ae18c099bb79b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced65e913e2443db827ae18c099bb79b" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44795 } } }
I20260807 22:45:17.876510 16872 sys_catalog.cc:458] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:17.876909 16879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:17.878846 16879 catalog_manager.cc:1347] Generated new cluster ID: 27f6459e7612408ab16dbcad2da6d86e
I20260807 22:45:17.878906 16879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 22:45:17 dist-test-slave-rgb1 krb5kdc[16786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:17.895195 16879 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:17.896034 16879 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:17.897135 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:17.875963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38954 (local address 127.16.98.190:44795)
0807 22:45:17.876780 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:17.876791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:17.876834 (+ 43us) server_negotiation.cc:407] Connection header received
0807 22:45:17.879461 (+ 2627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:17.879468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:17.879528 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:17.879675 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:17.882231 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:17.883676 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:17.885748 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:17.886537 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:17.886669 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:17.892130 (+ 5461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:17.892135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:17.892135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:17.892152 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:17.894641 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:17.895388 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:17.895391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:17.895392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:17.895506 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:17.895781 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:17.895782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:17.895783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:17.896036 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:17.896323 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:17.896679 (+ 356us) server_negotiation.cc:299] Negotiation successful
0807 22:45:17.896844 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":123,"threads_started":1}
I20260807 22:45:17.906489 16879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ced65e913e2443db827ae18c099bb79b: Generated new TSK 0
I20260807 22:45:17.906754 16879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:45:17.954421 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:44795
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:18.061956 16893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:18.062204 16893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:18.062235 16893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:18.063841 16893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:18.063966 16893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:18.064004 16893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:45:18.065953 16893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:44795
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:18.066246 16893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:18.066540 16893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:18.072757 16901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:18.072803 16899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:18.072803 16898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:18.073293 16893 server_base.cc:1034] running on GCE node
I20260807 22:45:18.073544 16893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:18.073918 16893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:18.075126 16893 hybrid_clock.cc:648] HybridClock initialized: now 1786142718075109 us; error 32 us; skew 500 ppm
Aug 07 22:45:18 dist-test-slave-rgb1 krb5kdc[16786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:18.078956 16893 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:45:18.079567 16893 webserver.cc:460] Webserver started at http://127.16.98.129:35025/ using document root <none> and password file <none>
I20260807 22:45:18.079854 16893 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:18.079933 16893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:18.080112 16893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:18.081506 16893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "08babc1cc3b44cb4a4ed4affd34fdf96"
format_stamp: "Formatted at 2026-08-07 22:45:18 on dist-test-slave-rgb1"
server_key: "26cb3f5f7dbd2ca1762472566635b4f7"
server_key_iv: "8778f3b4e710f0b692582b0b0a25037f"
server_key_version: "encryptionkey@0"
I20260807 22:45:18.081934 16893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "08babc1cc3b44cb4a4ed4affd34fdf96"
format_stamp: "Formatted at 2026-08-07 22:45:18 on dist-test-slave-rgb1"
server_key: "26cb3f5f7dbd2ca1762472566635b4f7"
server_key_iv: "8778f3b4e710f0b692582b0b0a25037f"
server_key_version: "encryptionkey@0"
I20260807 22:45:18.083600 16893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:45:18.084623 16908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:18.084916 16893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:18.085033 16893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "08babc1cc3b44cb4a4ed4affd34fdf96"
format_stamp: "Formatted at 2026-08-07 22:45:18 on dist-test-slave-rgb1"
server_key: "26cb3f5f7dbd2ca1762472566635b4f7"
server_key_iv: "8778f3b4e710f0b692582b0b0a25037f"
server_key_version: "encryptionkey@0"
I20260807 22:45:18.085100 16893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:18.109048 16893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:18.111976 16893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:18.112131 16893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:18.112365 16893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:18.112720 16893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:18.112759 16893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:18.112784 16893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:18.112799 16893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:18.120157 16893 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:38683
I20260807 22:45:18.120268 17023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:38683 every 8 connection(s)
I20260807 22:45:18.120671 16893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:45:18.121840 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 16893
I20260807 22:45:18.121953 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:45:18.122143 16778 external_mini_cluster.cc:1428] Setting key 0ce115755797068b5c0e587c4c1f9edd
Aug 07 22:45:18 dist-test-slave-rgb1 krb5kdc[16786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
I20260807 22:45:18.133186 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:18.121635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:47187 (local address 127.16.98.190:44795)
0807 22:45:18.121702 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:18.121705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:18.121929 (+ 224us) server_negotiation.cc:407] Connection header received
0807 22:45:18.122751 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:18.122752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:18.122767 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:18.122813 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:18.124306 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.125013 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.126160 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.126406 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.126505 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:18.129524 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:18.129525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:18.129526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:18.129535 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:18.131354 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:18.131948 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:18.131949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:18.131950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:18.132015 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:18.132371 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:18.132372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:18.132372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:18.132498 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:18.132622 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:18.132957 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 22:45:18.133109 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 22:45:18.133322 17026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:18.121569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:44795 (local address 127.16.98.129:47187)
0807 22:45:18.121794 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:18.121836 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:18.122629 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:18.122902 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:18.122907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:18.123904 (+ 997us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:18.124135 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:18.124142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.125145 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.125146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:18.126047 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:18.126050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.126521 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.126523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:18.126681 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:18.127333 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:18.127343 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:18.129400 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:18.131463 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:18.131474 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:18.131478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:18.131856 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:18.132150 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:18.132152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:18.132152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:18.132293 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:18.132635 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:18.132644 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:18.132805 (+ 161us) client_negotiation.cc:769] Sending connection context
0807 22:45:18.132930 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 22:45:18.133096 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":141,"threads_started":1}
I20260807 22:45:18.134228 17024 heartbeater.cc:347] Connected to a master server at 127.16.98.190:44795
I20260807 22:45:18.134379 17024 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:18.134613 17024 heartbeater.cc:511] Master 127.16.98.190:44795 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
I20260807 22:45:18.135421 16831 ts_manager.cc:194] Registered new tserver with Master: 08babc1cc3b44cb4a4ed4affd34fdf96 (127.16.98.129:38683)
I20260807 22:45:18.136471 16831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:47187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:45:18.159125 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:44795
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:18.256243 17029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:18.256486 17029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:18.256516 17029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:18.257843 17029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:18.257899 17029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:18.257942 17029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:45:18.259541 17029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:44795
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:18.259749 17029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:18.259961 17029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:18.266250 17035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:18.266254 17034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:18.266255 17037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:18.266273 17029 server_base.cc:1034] running on GCE node
I20260807 22:45:18.266741 17029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:18.266997 17029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:18.268159 17029 hybrid_clock.cc:648] HybridClock initialized: now 1786142718268149 us; error 36 us; skew 500 ppm
Aug 07 22:45:18 dist-test-slave-rgb1 krb5kdc[16786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:18.271199 17029 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:45:18.271935 17029 webserver.cc:460] Webserver started at http://127.16.98.130:41401/ using document root <none> and password file <none>
I20260807 22:45:18.272235 17029 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:18.272289 17029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:18.272480 17029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:18.273794 17029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "ab198ce9cfb041858967278cde3ad74d"
format_stamp: "Formatted at 2026-08-07 22:45:18 on dist-test-slave-rgb1"
server_key: "8f5ad4a5afe8b00aa668ca070425d5d3"
server_key_iv: "ebcb594adafe9326e6f9720a39109224"
server_key_version: "encryptionkey@0"
I20260807 22:45:18.274236 17029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "ab198ce9cfb041858967278cde3ad74d"
format_stamp: "Formatted at 2026-08-07 22:45:18 on dist-test-slave-rgb1"
server_key: "8f5ad4a5afe8b00aa668ca070425d5d3"
server_key_iv: "ebcb594adafe9326e6f9720a39109224"
server_key_version: "encryptionkey@0"
I20260807 22:45:18.276307 17029 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 22:45:18.277486 17044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:18.277834 17029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:18.277977 17029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "ab198ce9cfb041858967278cde3ad74d"
format_stamp: "Formatted at 2026-08-07 22:45:18 on dist-test-slave-rgb1"
server_key: "8f5ad4a5afe8b00aa668ca070425d5d3"
server_key_iv: "ebcb594adafe9326e6f9720a39109224"
server_key_version: "encryptionkey@0"
I20260807 22:45:18.278085 17029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:18.286761 17029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:18.289577 17029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:18.289775 17029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:18.290082 17029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:18.290443 17029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:18.290512 17029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:18.290565 17029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:18.290601 17029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:18.299365 17029 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:38211
I20260807 22:45:18.299434 17158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:38211 every 8 connection(s)
I20260807 22:45:18.299839 17029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:45:18.306123 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 17029
I20260807 22:45:18.306252 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:45:18.306443 16778 external_mini_cluster.cc:1428] Setting key a570fe8f85c29a208c42e02d2e0ffff9
Aug 07 22:45:18 dist-test-slave-rgb1 krb5kdc[16786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:18.312031 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:18.300738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:37049 (local address 127.16.98.190:44795)
0807 22:45:18.300808 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:18.300810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:18.301117 (+ 307us) server_negotiation.cc:407] Connection header received
0807 22:45:18.301674 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:18.301675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:18.301688 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:18.301768 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:18.303088 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.303684 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.304969 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.305199 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.305277 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:18.308429 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:18.308431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:18.308431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:18.308441 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:18.310183 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:18.310747 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:18.310748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:18.310749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:18.310816 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:18.311103 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:18.311104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:18.311105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:18.311262 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:18.311338 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:18.311760 (+ 422us) server_negotiation.cc:299] Negotiation successful
0807 22:45:18.311865 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 22:45:18.312444 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:18.300743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:44795 (local address 127.16.98.130:37049)
0807 22:45:18.300960 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:18.301002 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:18.301542 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:18.301869 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:18.301873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:18.302541 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:18.302892 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:18.302898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.303814 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.303815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:18.304793 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:18.304795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.305314 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.305315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:18.305464 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:18.306287 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:18.306298 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:18.308328 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:18.310354 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:18.310364 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:18.310367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:18.310640 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:18.310901 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:18.310902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:18.310902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:18.311021 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:18.311428 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:18.311437 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:18.311584 (+ 147us) client_negotiation.cc:769] Sending connection context
0807 22:45:18.311706 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 22:45:18.311954 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":105,"threads_started":1}
I20260807 22:45:18.313091 17159 heartbeater.cc:347] Connected to a master server at 127.16.98.190:44795
I20260807 22:45:18.313216 17159 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:18.313448 17159 heartbeater.cc:511] Master 127.16.98.190:44795 requested a full tablet report, sending...
I20260807 22:45:18.313939 16831 ts_manager.cc:194] Registered new tserver with Master: ab198ce9cfb041858967278cde3ad74d (127.16.98.130:38211)
I20260807 22:45:18.314383 16831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:37049
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:45:18.341480 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:44795
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:18.436668 17164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:18.436932 17164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:18.436966 17164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:18.438302 17164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:18.438350 17164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:18.438385 17164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:45:18.440291 17164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:44795
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:18.440613 17164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:18.440876 17164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:18.447217 17169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:18.447284 17164 server_base.cc:1034] running on GCE node
W20260807 22:45:18.447268 17172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:18.447211 17170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:18.447785 17164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:18.448120 17164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:18.449313 17164 hybrid_clock.cc:648] HybridClock initialized: now 1786142718449304 us; error 45 us; skew 500 ppm
Aug 07 22:45:18 dist-test-slave-rgb1 krb5kdc[16786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:18.452657 17164 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:45:18.453194 17164 webserver.cc:460] Webserver started at http://127.16.98.131:44283/ using document root <none> and password file <none>
I20260807 22:45:18.453474 17164 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:18.453554 17164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:18.453722 17164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:18.454999 17164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "0f87c2f97b00488b87d5cbd0ad8e803a"
format_stamp: "Formatted at 2026-08-07 22:45:18 on dist-test-slave-rgb1"
server_key: "0b8c303248bdaba40a5b9d66faf8740c"
server_key_iv: "5ff40ea765a4d1705c35fa625f5b5c0d"
server_key_version: "encryptionkey@0"
I20260807 22:45:18.455471 17164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "0f87c2f97b00488b87d5cbd0ad8e803a"
format_stamp: "Formatted at 2026-08-07 22:45:18 on dist-test-slave-rgb1"
server_key: "0b8c303248bdaba40a5b9d66faf8740c"
server_key_iv: "5ff40ea765a4d1705c35fa625f5b5c0d"
server_key_version: "encryptionkey@0"
I20260807 22:45:18.457726 17164 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.004s
I20260807 22:45:18.459224 17179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:18.459708 17164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:18.459810 17164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "0f87c2f97b00488b87d5cbd0ad8e803a"
format_stamp: "Formatted at 2026-08-07 22:45:18 on dist-test-slave-rgb1"
server_key: "0b8c303248bdaba40a5b9d66faf8740c"
server_key_iv: "5ff40ea765a4d1705c35fa625f5b5c0d"
server_key_version: "encryptionkey@0"
I20260807 22:45:18.459873 17164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:18.471863 17164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:18.474613 17164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:18.474817 17164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:18.475041 17164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:18.475512 17164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:18.475579 17164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:18.475623 17164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:18.475651 17164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:18.483217 17164 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:40531
I20260807 22:45:18.483451 17295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:40531 every 8 connection(s)
I20260807 22:45:18.483707 17164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:45:18.488718 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 17164
I20260807 22:45:18.488837 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:45:18.489058 16778 external_mini_cluster.cc:1428] Setting key 21a61a186297818e2071b74cd0d25e26
Aug 07 22:45:18 dist-test-slave-rgb1 krb5kdc[16786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142718, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:18.496536 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:18.484736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:41109 (local address 127.16.98.190:44795)
0807 22:45:18.484812 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:18.484814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:18.484987 (+ 173us) server_negotiation.cc:407] Connection header received
0807 22:45:18.485466 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:18.485468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:18.485479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:18.485565 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:18.486554 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.487119 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.488628 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.489090 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.489192 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:18.492460 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:18.492462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:18.492462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:18.492478 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:18.494236 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:18.494888 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:18.494889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:18.494890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:18.494953 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:18.495565 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:18.495567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:18.495568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:18.495784 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:18.495882 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:18.496349 (+ 467us) server_negotiation.cc:299] Negotiation successful
0807 22:45:18.496456 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 22:45:18.497001 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:18.484680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:44795 (local address 127.16.98.131:41109)
0807 22:45:18.484843 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:18.484887 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:18.485377 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:18.485617 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:18.485620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:18.486237 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:18.486434 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:18.486439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.487290 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.487291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:18.488436 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:18.488461 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.489195 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.489196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:18.489365 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:18.490451 (+ 1086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:18.490463 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:18.492346 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:18.494337 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:18.494344 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:18.494346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:18.494787 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:18.495153 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:18.495156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:18.495156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:18.495402 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:18.495912 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:18.495921 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:18.496106 (+ 185us) client_negotiation.cc:769] Sending connection context
0807 22:45:18.496242 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 22:45:18.496459 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260807 22:45:18.497666 17296 heartbeater.cc:347] Connected to a master server at 127.16.98.190:44795
I20260807 22:45:18.497767 17296 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:18.497947 17296 heartbeater.cc:511] Master 127.16.98.190:44795 requested a full tablet report, sending...
I20260807 22:45:18.498399 16831 ts_manager.cc:194] Registered new tserver with Master: 0f87c2f97b00488b87d5cbd0ad8e803a (127.16.98.131:40531)
I20260807 22:45:18.498765 16831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:41109
I20260807 22:45:18.502082 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 22:45:18 dist-test-slave-rgb1 krb5kdc[16786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 22:45:18 dist-test-slave-rgb1 krb5kdc[16786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142718, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.98.129@KRBTEST.COM
I20260807 22:45:18.530220 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:18.520900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35716 (local address 127.16.98.129:38683)
0807 22:45:18.521255 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:18.521260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:18.521271 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:45:18.521297 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:18.521299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:18.521314 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:18.521388 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:18.522204 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.522983 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.523825 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.524235 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.524304 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:18.526958 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:18.526968 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:18.526968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:18.526976 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:18.528812 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:18.529199 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:18.529202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:18.529202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:18.529263 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:18.529541 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:18.529542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:18.529542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:18.529708 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:18.529847 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:18.529953 (+ 106us) server_negotiation.cc:299] Negotiation successful
0807 22:45:18.530027 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":257,"threads_started":1}
W20260807 22:45:18.530637 16937 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35716
Aug 07 22:45:18 dist-test-slave-rgb1 krb5kdc[16786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 22:45:18 dist-test-slave-rgb1 krb5kdc[16786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.129@KRBTEST.COM
I20260807 22:45:18.558167 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:18.549173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35722 (local address 127.16.98.129:38683)
0807 22:45:18.549250 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:18.549253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:18.549302 (+ 49us) server_negotiation.cc:407] Connection header received
0807 22:45:18.549429 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:18.549430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:18.549442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:18.549492 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:18.550296 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.551054 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.551946 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:18.552174 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:18.552253 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:18.554970 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:18.554972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:18.554972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:18.554981 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:18.556758 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:18.557185 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:18.557187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:18.557187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:18.557252 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:18.557481 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:18.557482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:18.557482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:18.557602 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:18.557720 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:18.557900 (+ 180us) server_negotiation.cc:299] Negotiation successful
0807 22:45:18.557979 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 22:45:18.558794 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 16893
I20260807 22:45:18.566023 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 17029
I20260807 22:45:18.572130 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 17164
I20260807 22:45:18.578079 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 16799
2026-08-07T22:45:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2169 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:45:18 dist-test-slave-rgb1 krb5kdc[17310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:45:18 dist-test-slave-rgb1 krb5kdc[17310](info): set up 2 sockets
Aug 07 22:45:18 dist-test-slave-rgb1 krb5kdc[17310](info): commencing operation
krb5kdc: starting...
W20260807 22:45:19.708928 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:45:19 dist-test-slave-rgb1 krb5kdc[17310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:45:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:45:19.852480 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:36647
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:36647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:19.941756 17323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:19.941985 17323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:19.942008 17323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:19.943226 17323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:19.943277 17323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:19.943291 17323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:19.943305 17323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:19.943316 17323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:19.944909 17323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:36647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:36647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:19.945169 17323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:19.945425 17323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:19.950763 17328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:19.950793 17329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:19.950788 17331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:19.951227 17323 server_base.cc:1034] running on GCE node
I20260807 22:45:19.951498 17323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:19.951766 17323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:19.953019 17323 hybrid_clock.cc:648] HybridClock initialized: now 1786142719952921 us; error 33 us; skew 500 ppm
Aug 07 22:45:19 dist-test-slave-rgb1 krb5kdc[17310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142719, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:19.956077 17323 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:45:19.956686 17323 webserver.cc:460] Webserver started at http://127.16.98.190:35939/ using document root <none> and password file <none>
I20260807 22:45:19.957039 17323 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:19.957125 17323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:19.957293 17323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:19.958545 17323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "5df6e5994aaa4ba18e28d90ea9329ec3"
format_stamp: "Formatted at 2026-08-07 22:45:19 on dist-test-slave-rgb1"
server_key: "54bb31ff63dcfc238876358edee38cb0"
server_key_iv: "a9e9023856773d8ed489d12b6f6fc3f0"
server_key_version: "encryptionkey@0"
I20260807 22:45:19.958959 17323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "5df6e5994aaa4ba18e28d90ea9329ec3"
format_stamp: "Formatted at 2026-08-07 22:45:19 on dist-test-slave-rgb1"
server_key: "54bb31ff63dcfc238876358edee38cb0"
server_key_iv: "a9e9023856773d8ed489d12b6f6fc3f0"
server_key_version: "encryptionkey@0"
I20260807 22:45:19.960489 17323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260807 22:45:19.961613 17338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:19.961930 17323 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:19.962066 17323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "5df6e5994aaa4ba18e28d90ea9329ec3"
format_stamp: "Formatted at 2026-08-07 22:45:19 on dist-test-slave-rgb1"
server_key: "54bb31ff63dcfc238876358edee38cb0"
server_key_iv: "a9e9023856773d8ed489d12b6f6fc3f0"
server_key_version: "encryptionkey@0"
I20260807 22:45:19.962167 17323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:19.991354 17323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:19.993906 17323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:19.994107 17323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:19.998642 17323 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:36647
I20260807 22:45:19.998724 17391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:36647 every 8 connection(s)
I20260807 22:45:19.999121 17323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:20.000173 17392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:20.003090 17392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3: Bootstrap starting.
I20260807 22:45:20.003698 17392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:20.003964 17392 log.cc:824] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:20.004833 17392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3: No bootstrap required, opened a new log
I20260807 22:45:20.006495 17392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df6e5994aaa4ba18e28d90ea9329ec3" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36647 } }
I20260807 22:45:20.006666 17392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:20.006693 17392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df6e5994aaa4ba18e28d90ea9329ec3, State: Initialized, Role: FOLLOWER
I20260807 22:45:20.006810 17392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df6e5994aaa4ba18e28d90ea9329ec3" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36647 } }
I20260807 22:45:20.006860 17392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:20.006880 17392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:20.006907 17392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:20.007464 17392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df6e5994aaa4ba18e28d90ea9329ec3" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36647 } }
I20260807 22:45:20.007565 17392 leader_election.cc:302] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5df6e5994aaa4ba18e28d90ea9329ec3; no voters:
I20260807 22:45:20.007804 17392 leader_election.cc:288] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:20.007818 17395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:20.007968 17395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [term 1 LEADER]: Becoming Leader. State: Replica: 5df6e5994aaa4ba18e28d90ea9329ec3, State: Running, Role: LEADER
I20260807 22:45:20.008121 17392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:20.008091 17395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df6e5994aaa4ba18e28d90ea9329ec3" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36647 } }
I20260807 22:45:20.009232 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 17323
I20260807 22:45:20.009348 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:20.009553 16778 external_mini_cluster.cc:1428] Setting key 7e911bd549f6d609a25c1fa4f4c9a69a
I20260807 22:45:20.009094 17396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5df6e5994aaa4ba18e28d90ea9329ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df6e5994aaa4ba18e28d90ea9329ec3" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36647 } } }
I20260807 22:45:20.010160 17396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:20.009001 17397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5df6e5994aaa4ba18e28d90ea9329ec3. Latest consensus state: current_term: 1 leader_uuid: "5df6e5994aaa4ba18e28d90ea9329ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df6e5994aaa4ba18e28d90ea9329ec3" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36647 } } }
I20260807 22:45:20.010367 17397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3 [sys.catalog]: This master's current role is: LEADER
W20260807 22:45:20.010490 17410 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:45:20.010562 17410 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:45:20.010689 17413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:20.010867 17413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:20.012794 17413 catalog_manager.cc:1347] Generated new cluster ID: e747b2d012c34f389ce2d935a72226a7
I20260807 22:45:20.012859 17413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 22:45:20 dist-test-slave-rgb1 krb5kdc[17310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:20.024211 17413 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:20.024991 17413 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:20.026671 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.010203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.16.98.190:36647)
0807 22:45:20.011582 (+ 1379us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:20.011592 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:20.011631 (+ 39us) server_negotiation.cc:407] Connection header received
0807 22:45:20.011752 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:20.011759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:20.011835 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:20.011998 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:20.017022 (+ 5024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.018746 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.019617 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.020116 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.020277 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.022846 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:20.022852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:20.022852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:20.022871 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:20.024985 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.025403 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.025407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.025407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.025508 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.025742 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.025743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.025743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.026013 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:20.026164 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:20.026291 (+ 127us) server_negotiation.cc:299] Negotiation successful
0807 22:45:20.026454 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1328,"thread_start_us":71,"threads_started":1}
I20260807 22:45:20.033618 17413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5df6e5994aaa4ba18e28d90ea9329ec3: Generated new TSK 0
I20260807 22:45:20.033871 17413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:45:20.073107 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36647
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:20.188222 17418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:20.188470 17418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:20.188504 17418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:20.189944 17418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:20.190008 17418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:20.190040 17418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:45:20.191884 17418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36647
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:20.192183 17418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:20.192468 17418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 22:45:20.198926 17418 server_base.cc:1034] running on GCE node
W20260807 22:45:20.199190 17426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:20.199275 17423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:20.199247 17424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:20.199635 17418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:20.199954 17418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:20.201267 17418 hybrid_clock.cc:648] HybridClock initialized: now 1786142720201224 us; error 39 us; skew 500 ppm
Aug 07 22:45:20 dist-test-slave-rgb1 krb5kdc[17310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:20.205721 17418 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:45:20.206336 17418 webserver.cc:460] Webserver started at http://127.16.98.129:42335/ using document root <none> and password file <none>
I20260807 22:45:20.206635 17418 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:20.206712 17418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:20.206874 17418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:20.208103 17418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "a3c06955ba654a3a9bf44fe2678eeb54"
format_stamp: "Formatted at 2026-08-07 22:45:20 on dist-test-slave-rgb1"
server_key: "7042cfa0146904e6b5f963eb5459385e"
server_key_iv: "af800f372d71a600ddfff577734ed211"
server_key_version: "encryptionkey@0"
I20260807 22:45:20.208614 17418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "a3c06955ba654a3a9bf44fe2678eeb54"
format_stamp: "Formatted at 2026-08-07 22:45:20 on dist-test-slave-rgb1"
server_key: "7042cfa0146904e6b5f963eb5459385e"
server_key_iv: "af800f372d71a600ddfff577734ed211"
server_key_version: "encryptionkey@0"
I20260807 22:45:20.210299 17418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:20.211261 17433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:20.211615 17418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:20.211724 17418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "a3c06955ba654a3a9bf44fe2678eeb54"
format_stamp: "Formatted at 2026-08-07 22:45:20 on dist-test-slave-rgb1"
server_key: "7042cfa0146904e6b5f963eb5459385e"
server_key_iv: "af800f372d71a600ddfff577734ed211"
server_key_version: "encryptionkey@0"
I20260807 22:45:20.211790 17418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:20.231815 17418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:20.234575 17418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:20.234711 17418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:20.234987 17418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:20.235392 17418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:20.235457 17418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:20.235505 17418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:20.235543 17418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:20.242465 17418 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:37717
I20260807 22:45:20.242516 17547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:37717 every 8 connection(s)
I20260807 22:45:20.242921 17418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
Aug 07 22:45:20 dist-test-slave-rgb1 krb5kdc[17310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:20.252466 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 17418
I20260807 22:45:20.252625 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:45:20.252808 16778 external_mini_cluster.cc:1428] Setting key 5a68e58a3e432ecc9fd349c17e731274
I20260807 22:45:20.254050 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.243749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:46793 (local address 127.16.98.190:36647)
0807 22:45:20.243852 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:20.243854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:20.244085 (+ 231us) server_negotiation.cc:407] Connection header received
0807 22:45:20.245208 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:20.245210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:20.245221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:20.245297 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:20.246031 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.246684 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.247732 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.247929 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.247997 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.250592 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:20.250594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:20.250594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:20.250601 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:20.252215 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.252730 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.252731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.252732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.252805 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.253153 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.253154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.253154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.253328 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:20.253400 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:20.253830 (+ 430us) server_negotiation.cc:299] Negotiation successful
0807 22:45:20.253940 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 22:45:20.254156 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.243725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36647 (local address 127.16.98.129:46793)
0807 22:45:20.243916 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:20.243961 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:20.245096 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:20.245355 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:20.245359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:20.245641 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:20.245856 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.245860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.246787 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.246788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.247633 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.247636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.248060 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.248062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.248199 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.248903 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:20.248912 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:20.250483 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:20.252322 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:20.252329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:20.252331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:20.252656 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:20.252882 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:20.252883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:20.252884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:20.253010 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:20.253428 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:20.253435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:20.253669 (+ 234us) client_negotiation.cc:769] Sending connection context
0807 22:45:20.253765 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 22:45:20.253914 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":99,"threads_started":1}
I20260807 22:45:20.254776 17548 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36647
I20260807 22:45:20.254905 17548 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:20.255127 17548 heartbeater.cc:511] Master 127.16.98.190:36647 requested a full tablet report, sending...
I20260807 22:45:20.255760 17355 ts_manager.cc:194] Registered new tserver with Master: a3c06955ba654a3a9bf44fe2678eeb54 (127.16.98.129:37717)
I20260807 22:45:20.256424 17355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:46793
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:45:20.288468 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36647
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:20.379177 17553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:20.379419 17553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:20.379449 17553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:20.380851 17553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:20.380903 17553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:20.380936 17553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:45:20.382651 17553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36647
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:20.382941 17553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:20.383229 17553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:20.389014 17561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:20.389101 17559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:20.389252 17553 server_base.cc:1034] running on GCE node
W20260807 22:45:20.389101 17558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:20.389588 17553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:20.389858 17553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:20.391086 17553 hybrid_clock.cc:648] HybridClock initialized: now 1786142720391072 us; error 33 us; skew 500 ppm
Aug 07 22:45:20 dist-test-slave-rgb1 krb5kdc[17310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:20.394178 17553 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:45:20.394662 17553 webserver.cc:460] Webserver started at http://127.16.98.130:36021/ using document root <none> and password file <none>
I20260807 22:45:20.394910 17553 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:20.394982 17553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:20.395139 17553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:20.396325 17553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "8cbfb8e1a0b54d0daa5372df9755d523"
format_stamp: "Formatted at 2026-08-07 22:45:20 on dist-test-slave-rgb1"
server_key: "dfec7b71b6a18d41cb73099e0290c83c"
server_key_iv: "525e8ac7dfad2e55c7c2001cd5a1bafb"
server_key_version: "encryptionkey@0"
I20260807 22:45:20.396741 17553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "8cbfb8e1a0b54d0daa5372df9755d523"
format_stamp: "Formatted at 2026-08-07 22:45:20 on dist-test-slave-rgb1"
server_key: "dfec7b71b6a18d41cb73099e0290c83c"
server_key_iv: "525e8ac7dfad2e55c7c2001cd5a1bafb"
server_key_version: "encryptionkey@0"
I20260807 22:45:20.398198 17553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:20.399156 17568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:20.399478 17553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:20.399669 17553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "8cbfb8e1a0b54d0daa5372df9755d523"
format_stamp: "Formatted at 2026-08-07 22:45:20 on dist-test-slave-rgb1"
server_key: "dfec7b71b6a18d41cb73099e0290c83c"
server_key_iv: "525e8ac7dfad2e55c7c2001cd5a1bafb"
server_key_version: "encryptionkey@0"
I20260807 22:45:20.399775 17553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:20.426087 17553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:20.428615 17553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:20.428769 17553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:20.428980 17553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:20.429294 17553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:20.429337 17553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:20.429361 17553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:20.429374 17553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:20.435698 17553 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:37521
I20260807 22:45:20.435819 17682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:37521 every 8 connection(s)
I20260807 22:45:20.436151 17553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
Aug 07 22:45:20 dist-test-slave-rgb1 krb5kdc[17310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:20.445127 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 17553
I20260807 22:45:20.445261 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:45:20.445459 16778 external_mini_cluster.cc:1428] Setting key f5c6515b9c8ba76be15923b428bae216
I20260807 22:45:20.447782 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.437107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36647 (local address 127.16.98.130:47829)
0807 22:45:20.437348 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:20.437388 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:20.438327 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:20.438770 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:20.438781 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:20.439100 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:20.439300 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.439305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.440149 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.440149 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.441183 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.441186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.441649 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.441650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.441793 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.442388 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:20.442397 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:20.444064 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:20.446177 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:20.446186 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:20.446189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:20.446461 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:20.446722 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:20.446723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:20.446723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:20.446847 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:20.447228 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:20.447235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:20.447365 (+ 130us) client_negotiation.cc:769] Sending connection context
0807 22:45:20.447470 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 22:45:20.447609 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":116,"threads_started":1}
I20260807 22:45:20.447791 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.437131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:47829 (local address 127.16.98.190:36647)
0807 22:45:20.437213 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:20.437216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:20.437488 (+ 272us) server_negotiation.cc:407] Connection header received
0807 22:45:20.438440 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:20.438441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:20.438456 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:20.438552 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:20.439434 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.440048 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.441310 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.441529 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.441593 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.444195 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:20.444197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:20.444197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:20.444206 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:20.446012 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.446552 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.446553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.446554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.446624 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.446930 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.446931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.446932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.447115 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:20.447195 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:20.447517 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 22:45:20.447632 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 22:45:20.448513 17683 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36647
I20260807 22:45:20.448674 17683 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:20.448909 17683 heartbeater.cc:511] Master 127.16.98.190:36647 requested a full tablet report, sending...
I20260807 22:45:20.449373 17355 ts_manager.cc:194] Registered new tserver with Master: 8cbfb8e1a0b54d0daa5372df9755d523 (127.16.98.130:37521)
I20260807 22:45:20.449765 17355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:47829
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:45:20.481348 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36647
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:20.573984 17688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:20.574216 17688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:20.574244 17688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:20.575690 17688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:20.575755 17688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:20.575788 17688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:45:20.577581 17688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36647
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:20.577793 17688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:20.577987 17688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:20.583319 17697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:20.583303 17695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:20.583299 17694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:20.583688 17688 server_base.cc:1034] running on GCE node
I20260807 22:45:20.583951 17688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:20.584255 17688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:20.585415 17688 hybrid_clock.cc:648] HybridClock initialized: now 1786142720585401 us; error 32 us; skew 500 ppm
Aug 07 22:45:20 dist-test-slave-rgb1 krb5kdc[17310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:20.588264 17688 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:45:20.588747 17688 webserver.cc:460] Webserver started at http://127.16.98.131:40563/ using document root <none> and password file <none>
I20260807 22:45:20.589010 17688 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:20.589079 17688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:20.589233 17688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:20.590301 17688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "cf7d8079ff9c4b0e9412313ae9a65adc"
format_stamp: "Formatted at 2026-08-07 22:45:20 on dist-test-slave-rgb1"
server_key: "1d26912ab524e2bc2d39c3e6b18580a8"
server_key_iv: "dcbd5122342eeec7c4f8f6c9627e0aef"
server_key_version: "encryptionkey@0"
I20260807 22:45:20.590704 17688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "cf7d8079ff9c4b0e9412313ae9a65adc"
format_stamp: "Formatted at 2026-08-07 22:45:20 on dist-test-slave-rgb1"
server_key: "1d26912ab524e2bc2d39c3e6b18580a8"
server_key_iv: "dcbd5122342eeec7c4f8f6c9627e0aef"
server_key_version: "encryptionkey@0"
I20260807 22:45:20.592144 17688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:45:20.593010 17704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:20.593187 17688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 22:45:20.593278 17688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "cf7d8079ff9c4b0e9412313ae9a65adc"
format_stamp: "Formatted at 2026-08-07 22:45:20 on dist-test-slave-rgb1"
server_key: "1d26912ab524e2bc2d39c3e6b18580a8"
server_key_iv: "dcbd5122342eeec7c4f8f6c9627e0aef"
server_key_version: "encryptionkey@0"
I20260807 22:45:20.593362 17688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:20.603308 17688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:20.605738 17688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:20.605911 17688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:20.606178 17688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:20.606544 17688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:20.606601 17688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:20.606645 17688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:20.606693 17688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:20.614009 17688 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:46207
I20260807 22:45:20.614078 17820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:46207 every 8 connection(s)
I20260807 22:45:20.614527 17688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:45:20.617875 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 17688
I20260807 22:45:20.617975 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:45:20.618171 16778 external_mini_cluster.cc:1428] Setting key 370cbb009f0ec8960713e9cc9bafaa82
Aug 07 22:45:20 dist-test-slave-rgb1 krb5kdc[17310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142720, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:20.626260 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.615524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:50793 (local address 127.16.98.190:36647)
0807 22:45:20.615576 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:20.615579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:20.615840 (+ 261us) server_negotiation.cc:407] Connection header received
0807 22:45:20.617061 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:20.617062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:20.617078 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:20.617136 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:20.617855 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.618653 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.619960 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.620244 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.620311 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.622893 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:20.622895 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:20.622895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:20.622903 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:20.624735 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.625161 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.625162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.625162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.625224 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.625578 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.625579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.625579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.625704 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:20.625772 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:20.626116 (+ 344us) server_negotiation.cc:299] Negotiation successful
0807 22:45:20.626191 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 22:45:20.626508 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.615540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36647 (local address 127.16.98.131:50793)
0807 22:45:20.615699 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:20.615740 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:20.616921 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:20.617218 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:20.617222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:20.617518 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:20.617751 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.617755 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.618763 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.618764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.619874 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.619877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.620381 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.620382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.620511 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.621185 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:20.621195 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:20.622773 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:20.624819 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:20.624826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:20.624828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:20.625083 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:20.625317 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:20.625318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:20.625319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:20.625480 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:20.625806 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:20.625812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:20.625959 (+ 147us) client_negotiation.cc:769] Sending connection context
0807 22:45:20.626064 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 22:45:20.626210 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260807 22:45:20.627224 17821 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36647
I20260807 22:45:20.627353 17821 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:20.627645 17821 heartbeater.cc:511] Master 127.16.98.190:36647 requested a full tablet report, sending...
I20260807 22:45:20.628157 17355 ts_manager.cc:194] Registered new tserver with Master: cf7d8079ff9c4b0e9412313ae9a65adc (127.16.98.131:46207)
I20260807 22:45:20.628513 17355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:50793
I20260807 22:45:20.630106 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:45:20.638271 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.631774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44360 (local address 127.16.98.190:36647)
0807 22:45:20.631826 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:20.631828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:20.631859 (+ 31us) server_negotiation.cc:407] Connection header received
0807 22:45:20.631916 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:20.631916 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:20.631925 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:20.631985 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:20.632674 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.633222 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.633953 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.634169 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.634232 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.635578 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:20.635579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:20.635580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:20.635586 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:20.637107 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.637541 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.637542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.637542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.637599 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.637818 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.637818 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.637818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.637961 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:20.638038 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:20.638142 (+ 104us) server_negotiation.cc:299] Negotiation successful
0807 22:45:20.638205 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 22:45:20.640851 17355 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:45:20.642596 17355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:20.650056 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.645381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.16.98.130:37521)
0807 22:45:20.645798 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:20.645802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:20.646082 (+ 280us) server_negotiation.cc:407] Connection header received
0807 22:45:20.646146 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:20.646149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:20.646168 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:20.646261 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:20.646630 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.647729 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.649076 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.649618 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.649697 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.649754 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:20.649921 (+ 167us) server_negotiation.cc:299] Negotiation successful
0807 22:45:20.649969 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":80,"threads_started":1}
I20260807 22:45:20.650063 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.645181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:37521 (local address 127.0.0.1:43276)
0807 22:45:20.645812 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:20.645821 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:20.645873 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:20.646377 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:20.646381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:20.646386 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:20.646541 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.646543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.647849 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.647850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.648986 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.648989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.649710 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.649711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.649798 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.649814 (+ 16us) client_negotiation.cc:769] Sending connection context
0807 22:45:20.649898 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 22:45:20.649958 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":75,"thread_start_us":47,"threads_started":1}
I20260807 22:45:20.650537 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.645181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:46207 (local address 127.0.0.1:47628)
0807 22:45:20.645586 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:20.645635 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:20.645717 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:20.646377 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:20.646381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:20.646383 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:20.646496 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.646499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.647650 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.647651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.649114 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.649117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.650267 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.650268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.650332 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.650336 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:20.650393 (+ 57us) client_negotiation.cc:240] Negotiation successful
0807 22:45:20.650438 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260807 22:45:20.650556 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.645335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47628 (local address 127.16.98.131:46207)
0807 22:45:20.646103 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:20.646108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:20.646117 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:45:20.646152 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:20.646155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:20.646170 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:20.646261 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:20.646593 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.647534 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.649237 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.650083 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.650164 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.650228 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:20.650428 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 22:45:20.650474 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":122,"threads_started":1}
I20260807 22:45:20.650784 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.645423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:37717 (local address 127.0.0.1:47114)
0807 22:45:20.646616 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:20.646627 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:20.646675 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:20.646899 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:20.646900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:20.646903 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:20.646950 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.646952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.648194 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.648195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.649494 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.649498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.650529 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.650530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.650596 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.650600 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:20.650659 (+ 59us) client_negotiation.cc:240] Negotiation successful
0807 22:45:20.650709 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1173,"thread_start_us":81,"threads_started":1}
I20260807 22:45:20.651021 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.645564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47114 (local address 127.16.98.129:37717)
0807 22:45:20.645749 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:20.645754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:20.646736 (+ 982us) server_negotiation.cc:407] Connection header received
0807 22:45:20.646782 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:20.646784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:20.646800 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:20.646876 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:20.647055 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.648085 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.649591 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.650329 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.650429 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.650492 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:20.650883 (+ 391us) server_negotiation.cc:299] Negotiation successful
0807 22:45:20.650939 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260807 22:45:20.651208 17617 tablet_service.cc:1467] Processing CreateTablet for tablet 8a63c2585c0e4a99932b0781b3bdf756 (DEFAULT_TABLE table=test-table [id=acab0720b1db42348d95de7e55dd0b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:20.651608 17617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a63c2585c0e4a99932b0781b3bdf756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:20.653409 17754 tablet_service.cc:1467] Processing CreateTablet for tablet 8a63c2585c0e4a99932b0781b3bdf756 (DEFAULT_TABLE table=test-table [id=acab0720b1db42348d95de7e55dd0b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:20.653646 17482 tablet_service.cc:1467] Processing CreateTablet for tablet 8a63c2585c0e4a99932b0781b3bdf756 (DEFAULT_TABLE table=test-table [id=acab0720b1db42348d95de7e55dd0b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:20.653843 17754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a63c2585c0e4a99932b0781b3bdf756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:20.654026 17482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a63c2585c0e4a99932b0781b3bdf756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:20.655268 17838 tablet_bootstrap.cc:492] T 8a63c2585c0e4a99932b0781b3bdf756 P 8cbfb8e1a0b54d0daa5372df9755d523: Bootstrap starting.
I20260807 22:45:20.656169 17838 tablet_bootstrap.cc:654] T 8a63c2585c0e4a99932b0781b3bdf756 P 8cbfb8e1a0b54d0daa5372df9755d523: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:20.656893 17838 log.cc:824] T 8a63c2585c0e4a99932b0781b3bdf756 P 8cbfb8e1a0b54d0daa5372df9755d523: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:20.657577 17840 tablet_bootstrap.cc:492] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc: Bootstrap starting.
I20260807 22:45:20.657856 17838 tablet_bootstrap.cc:492] T 8a63c2585c0e4a99932b0781b3bdf756 P 8cbfb8e1a0b54d0daa5372df9755d523: No bootstrap required, opened a new log
I20260807 22:45:20.657922 17838 ts_tablet_manager.cc:1397] T 8a63c2585c0e4a99932b0781b3bdf756 P 8cbfb8e1a0b54d0daa5372df9755d523: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:45:20.658322 17840 tablet_bootstrap.cc:654] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:20.658879 17840 log.cc:824] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:20.659740 17840 tablet_bootstrap.cc:492] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc: No bootstrap required, opened a new log
I20260807 22:45:20.659924 17840 ts_tablet_manager.cc:1397] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:45:20.659914 17838 raft_consensus.cc:348] T 8a63c2585c0e4a99932b0781b3bdf756 P 8cbfb8e1a0b54d0daa5372df9755d523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7d8079ff9c4b0e9412313ae9a65adc" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 46207 } } peers { permanent_uuid: "8cbfb8e1a0b54d0daa5372df9755d523" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37521 } } peers { permanent_uuid: "a3c06955ba654a3a9bf44fe2678eeb54" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37717 } }
I20260807 22:45:20.660115 17838 raft_consensus.cc:374] T 8a63c2585c0e4a99932b0781b3bdf756 P 8cbfb8e1a0b54d0daa5372df9755d523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:20.660163 17838 raft_consensus.cc:729] T 8a63c2585c0e4a99932b0781b3bdf756 P 8cbfb8e1a0b54d0daa5372df9755d523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cbfb8e1a0b54d0daa5372df9755d523, State: Initialized, Role: FOLLOWER
I20260807 22:45:20.660305 17838 consensus_queue.cc:260] T 8a63c2585c0e4a99932b0781b3bdf756 P 8cbfb8e1a0b54d0daa5372df9755d523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7d8079ff9c4b0e9412313ae9a65adc" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 46207 } } peers { permanent_uuid: "8cbfb8e1a0b54d0daa5372df9755d523" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37521 } } peers { permanent_uuid: "a3c06955ba654a3a9bf44fe2678eeb54" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37717 } }
I20260807 22:45:20.660662 17683 heartbeater.cc:503] Master 127.16.98.190:36647 was elected leader, sending a full tablet report...
I20260807 22:45:20.660770 17841 tablet_bootstrap.cc:492] T 8a63c2585c0e4a99932b0781b3bdf756 P a3c06955ba654a3a9bf44fe2678eeb54: Bootstrap starting.
I20260807 22:45:20.660992 17838 ts_tablet_manager.cc:1428] T 8a63c2585c0e4a99932b0781b3bdf756 P 8cbfb8e1a0b54d0daa5372df9755d523: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:45:20.661442 17841 tablet_bootstrap.cc:654] T 8a63c2585c0e4a99932b0781b3bdf756 P a3c06955ba654a3a9bf44fe2678eeb54: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:20.661934 17841 log.cc:824] T 8a63c2585c0e4a99932b0781b3bdf756 P a3c06955ba654a3a9bf44fe2678eeb54: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:20.662453 17840 raft_consensus.cc:348] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7d8079ff9c4b0e9412313ae9a65adc" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 46207 } } peers { permanent_uuid: "8cbfb8e1a0b54d0daa5372df9755d523" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37521 } } peers { permanent_uuid: "a3c06955ba654a3a9bf44fe2678eeb54" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37717 } }
I20260807 22:45:20.662703 17840 raft_consensus.cc:374] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:20.662748 17840 raft_consensus.cc:729] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7d8079ff9c4b0e9412313ae9a65adc, State: Initialized, Role: FOLLOWER
I20260807 22:45:20.662873 17841 tablet_bootstrap.cc:492] T 8a63c2585c0e4a99932b0781b3bdf756 P a3c06955ba654a3a9bf44fe2678eeb54: No bootstrap required, opened a new log
I20260807 22:45:20.662879 17840 consensus_queue.cc:260] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7d8079ff9c4b0e9412313ae9a65adc" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 46207 } } peers { permanent_uuid: "8cbfb8e1a0b54d0daa5372df9755d523" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37521 } } peers { permanent_uuid: "a3c06955ba654a3a9bf44fe2678eeb54" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37717 } }
I20260807 22:45:20.662953 17841 ts_tablet_manager.cc:1397] T 8a63c2585c0e4a99932b0781b3bdf756 P a3c06955ba654a3a9bf44fe2678eeb54: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:45:20.663228 17840 ts_tablet_manager.cc:1428] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260807 22:45:20.663305 17821 heartbeater.cc:503] Master 127.16.98.190:36647 was elected leader, sending a full tablet report...
I20260807 22:45:20.665202 17841 raft_consensus.cc:348] T 8a63c2585c0e4a99932b0781b3bdf756 P a3c06955ba654a3a9bf44fe2678eeb54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7d8079ff9c4b0e9412313ae9a65adc" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 46207 } } peers { permanent_uuid: "8cbfb8e1a0b54d0daa5372df9755d523" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37521 } } peers { permanent_uuid: "a3c06955ba654a3a9bf44fe2678eeb54" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37717 } }
I20260807 22:45:20.665354 17841 raft_consensus.cc:374] T 8a63c2585c0e4a99932b0781b3bdf756 P a3c06955ba654a3a9bf44fe2678eeb54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:20.665381 17841 raft_consensus.cc:729] T 8a63c2585c0e4a99932b0781b3bdf756 P a3c06955ba654a3a9bf44fe2678eeb54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c06955ba654a3a9bf44fe2678eeb54, State: Initialized, Role: FOLLOWER
I20260807 22:45:20.665503 17841 consensus_queue.cc:260] T 8a63c2585c0e4a99932b0781b3bdf756 P a3c06955ba654a3a9bf44fe2678eeb54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7d8079ff9c4b0e9412313ae9a65adc" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 46207 } } peers { permanent_uuid: "8cbfb8e1a0b54d0daa5372df9755d523" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37521 } } peers { permanent_uuid: "a3c06955ba654a3a9bf44fe2678eeb54" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37717 } }
I20260807 22:45:20.665827 17841 ts_tablet_manager.cc:1428] T 8a63c2585c0e4a99932b0781b3bdf756 P a3c06955ba654a3a9bf44fe2678eeb54: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:45:20.665938 17548 heartbeater.cc:503] Master 127.16.98.190:36647 was elected leader, sending a full tablet report...
W20260807 22:45:20.686825 17684 tablet.cc:2369] T 8a63c2585c0e4a99932b0781b3bdf756 P 8cbfb8e1a0b54d0daa5372df9755d523: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:45:20.743779 17549 tablet.cc:2369] T 8a63c2585c0e4a99932b0781b3bdf756 P a3c06955ba654a3a9bf44fe2678eeb54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:45:20.769120 17845 raft_consensus.cc:482] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:20.769263 17845 raft_consensus.cc:504] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7d8079ff9c4b0e9412313ae9a65adc" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 46207 } } peers { permanent_uuid: "8cbfb8e1a0b54d0daa5372df9755d523" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37521 } } peers { permanent_uuid: "a3c06955ba654a3a9bf44fe2678eeb54" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37717 } }
I20260807 22:45:20.769804 17845 leader_election.cc:288] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cbfb8e1a0b54d0daa5372df9755d523 (127.16.98.130:37521), a3c06955ba654a3a9bf44fe2678eeb54 (127.16.98.129:37717)
I20260807 22:45:20.773295 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.770208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:60011 (local address 127.16.98.129:37717)
0807 22:45:20.770260 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:20.770262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:20.770382 (+ 120us) server_negotiation.cc:407] Connection header received
0807 22:45:20.770484 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:20.770485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:20.770494 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:20.770694 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:20.770867 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.771383 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.772459 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.772878 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.772967 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.772979 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:20.773171 (+ 192us) server_negotiation.cc:299] Negotiation successful
0807 22:45:20.773214 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 22:45:20.773321 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.770113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:37717 (local address 127.16.98.131:60011)
0807 22:45:20.770289 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:20.770300 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:20.770393 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:20.770695 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:20.770697 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:20.770699 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:20.770785 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.770787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.771495 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.771496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.772352 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.772354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.772990 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.772991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.773078 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.773082 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:20.773150 (+ 68us) client_negotiation.cc:240] Negotiation successful
0807 22:45:20.773194 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":103,"threads_started":1}
I20260807 22:45:20.773319 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.770166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:57887 (local address 127.16.98.130:37521)
0807 22:45:20.770213 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:20.770216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:20.770227 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:45:20.770298 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:20.770299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:20.770311 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:20.770401 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:20.770591 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.771232 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.772379 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.772891 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.772991 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.773000 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:20.773196 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 22:45:20.773240 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 22:45:20.773322 17823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.770081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:37521 (local address 127.16.98.131:57887)
0807 22:45:20.770148 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:20.770166 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:20.770219 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:20.770406 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:20.770407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:20.770409 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:20.770507 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.770509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.771335 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.771337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.772268 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:20.772271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.772987 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.772988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:20.773111 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.773114 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:45:20.773189 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 22:45:20.773232 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":82}
I20260807 22:45:20.773728 17502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a63c2585c0e4a99932b0781b3bdf756" candidate_uuid: "cf7d8079ff9c4b0e9412313ae9a65adc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c06955ba654a3a9bf44fe2678eeb54" is_pre_election: true
I20260807 22:45:20.773908 17502 raft_consensus.cc:2454] T 8a63c2585c0e4a99932b0781b3bdf756 P a3c06955ba654a3a9bf44fe2678eeb54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf7d8079ff9c4b0e9412313ae9a65adc in term 0.
I20260807 22:45:20.773917 17637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a63c2585c0e4a99932b0781b3bdf756" candidate_uuid: "cf7d8079ff9c4b0e9412313ae9a65adc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cbfb8e1a0b54d0daa5372df9755d523" is_pre_election: true
I20260807 22:45:20.774052 17637 raft_consensus.cc:2454] T 8a63c2585c0e4a99932b0781b3bdf756 P 8cbfb8e1a0b54d0daa5372df9755d523 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf7d8079ff9c4b0e9412313ae9a65adc in term 0.
I20260807 22:45:20.774293 17708 leader_election.cc:302] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cbfb8e1a0b54d0daa5372df9755d523, cf7d8079ff9c4b0e9412313ae9a65adc; no voters:
I20260807 22:45:20.774490 17845 raft_consensus.cc:2781] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:20.774555 17845 raft_consensus.cc:482] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:20.774581 17845 raft_consensus.cc:3037] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:20.775350 17845 raft_consensus.cc:504] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7d8079ff9c4b0e9412313ae9a65adc" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 46207 } } peers { permanent_uuid: "8cbfb8e1a0b54d0daa5372df9755d523" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37521 } } peers { permanent_uuid: "a3c06955ba654a3a9bf44fe2678eeb54" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37717 } }
I20260807 22:45:20.775588 17845 leader_election.cc:288] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [CANDIDATE]: Term 1 election: Requested vote from peers 8cbfb8e1a0b54d0daa5372df9755d523 (127.16.98.130:37521), a3c06955ba654a3a9bf44fe2678eeb54 (127.16.98.129:37717)
I20260807 22:45:20.775724 17637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a63c2585c0e4a99932b0781b3bdf756" candidate_uuid: "cf7d8079ff9c4b0e9412313ae9a65adc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cbfb8e1a0b54d0daa5372df9755d523"
I20260807 22:45:20.775808 17637 raft_consensus.cc:3037] T 8a63c2585c0e4a99932b0781b3bdf756 P 8cbfb8e1a0b54d0daa5372df9755d523 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:20.775791 17502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a63c2585c0e4a99932b0781b3bdf756" candidate_uuid: "cf7d8079ff9c4b0e9412313ae9a65adc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c06955ba654a3a9bf44fe2678eeb54"
I20260807 22:45:20.775974 17502 raft_consensus.cc:3037] T 8a63c2585c0e4a99932b0781b3bdf756 P a3c06955ba654a3a9bf44fe2678eeb54 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:20.776716 17637 raft_consensus.cc:2454] T 8a63c2585c0e4a99932b0781b3bdf756 P 8cbfb8e1a0b54d0daa5372df9755d523 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf7d8079ff9c4b0e9412313ae9a65adc in term 1.
I20260807 22:45:20.776890 17708 leader_election.cc:302] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cbfb8e1a0b54d0daa5372df9755d523, cf7d8079ff9c4b0e9412313ae9a65adc; no voters:
I20260807 22:45:20.776916 17502 raft_consensus.cc:2454] T 8a63c2585c0e4a99932b0781b3bdf756 P a3c06955ba654a3a9bf44fe2678eeb54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf7d8079ff9c4b0e9412313ae9a65adc in term 1.
I20260807 22:45:20.776997 17845 raft_consensus.cc:2781] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:20.777175 17845 raft_consensus.cc:686] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [term 1 LEADER]: Becoming Leader. State: Replica: cf7d8079ff9c4b0e9412313ae9a65adc, State: Running, Role: LEADER
I20260807 22:45:20.777293 17845 consensus_queue.cc:237] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7d8079ff9c4b0e9412313ae9a65adc" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 46207 } } peers { permanent_uuid: "8cbfb8e1a0b54d0daa5372df9755d523" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37521 } } peers { permanent_uuid: "a3c06955ba654a3a9bf44fe2678eeb54" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37717 } }
I20260807 22:45:20.778187 17355 catalog_manager.cc:5495] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc reported cstate change: term changed from 0 to 1, leader changed from <none> to cf7d8079ff9c4b0e9412313ae9a65adc (127.16.98.131). New cstate: current_term: 1 leader_uuid: "cf7d8079ff9c4b0e9412313ae9a65adc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf7d8079ff9c4b0e9412313ae9a65adc" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 46207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cbfb8e1a0b54d0daa5372df9755d523" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3c06955ba654a3a9bf44fe2678eeb54" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37717 } health_report { overall_health: UNKNOWN } } }
Aug 07 22:45:20 dist-test-slave-rgb1 krb5kdc[17310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.129@KRBTEST.COM
I20260807 22:45:20.808851 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.798233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.16.98.129:37717)
0807 22:45:20.798322 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:20.798325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:20.798336 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:45:20.798404 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:20.798405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:20.798414 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:20.798472 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:20.799088 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.799582 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.800420 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.800691 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.800750 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.803188 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:20.803198 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:20.803199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:20.803205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:20.805321 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.805768 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.805771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.805772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.807812 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.808074 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.808075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.808076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.808309 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:20.808485 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:20.808636 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 22:45:20.808729 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 22:45:20 dist-test-slave-rgb1 krb5kdc[17310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 22:45:20 dist-test-slave-rgb1 krb5kdc[17310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.98.129@KRBTEST.COM
I20260807 22:45:20.838914 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.830680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.16.98.129:37717)
0807 22:45:20.830778 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:20.830782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:20.830895 (+ 113us) server_negotiation.cc:407] Connection header received
0807 22:45:20.830982 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:20.830983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:20.830993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:20.831053 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:20.831747 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.832255 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.833193 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.833411 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.833499 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.835873 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:20.835875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:20.835875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:20.835883 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:20.837559 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.837928 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.837930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.837930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.837992 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.838214 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.838215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.838216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.838326 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:20.838434 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:20.838629 (+ 195us) server_negotiation.cc:299] Negotiation successful
0807 22:45:20.838796 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 22:45:20 dist-test-slave-rgb1 krb5kdc[17310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 22:45:20 dist-test-slave-rgb1 krb5kdc[17310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.129@KRBTEST.COM
W20260807 22:45:20.865211 17822 tablet.cc:2369] T 8a63c2585c0e4a99932b0781b3bdf756 P cf7d8079ff9c4b0e9412313ae9a65adc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:45:20.865496 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:20.856928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.16.98.129:37717)
0807 22:45:20.857063 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:20.857066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:20.857096 (+ 30us) server_negotiation.cc:407] Connection header received
0807 22:45:20.857205 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:20.857206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:20.857217 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:20.857288 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:20.857917 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.858416 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.859461 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:20.859682 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:20.859779 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:20.862517 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:20.862519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:20.862524 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:20.862533 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:20.863991 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.864423 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.864424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.864425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.864488 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:20.864745 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:20.864746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:20.864746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:20.864858 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:20.864989 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:20.865207 (+ 218us) server_negotiation.cc:299] Negotiation successful
0807 22:45:20.865340 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260807 22:45:20.866118 17462 tablet_service.cc:2908] Rejecting scan request for tablet 8a63c2585c0e4a99932b0781b3bdf756: Uninitialized: clean time has not yet been initialized
I20260807 22:45:20.867116 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 17418
I20260807 22:45:20.874174 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 17553
I20260807 22:45:20.880385 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 17688
I20260807 22:45:20.885995 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 17323
2026-08-07T22:45:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2306 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:45:20 dist-test-slave-rgb1 krb5kdc[17862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:45:20 dist-test-slave-rgb1 krb5kdc[17862](info): set up 2 sockets
Aug 07 22:45:20 dist-test-slave-rgb1 krb5kdc[17862](info): commencing operation
krb5kdc: starting...
W20260807 22:45:22.009536 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:45:22 dist-test-slave-rgb1 krb5kdc[17862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:45:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:45:22.153048 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:36425
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:42877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:36425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:22.239032 17876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:22.239276 17876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:22.239315 17876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:22.240492 17876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:22.240566 17876 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 22:45:22.240621 17876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:22.240650 17876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:22.240693 17876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:22.240720 17876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:22.242242 17876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:36425
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:36425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:22.242527 17876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:22.242779 17876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:22.248245 17882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:22.248262 17876 server_base.cc:1034] running on GCE node
W20260807 22:45:22.248243 17884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:22.248252 17881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:22.248737 17876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:22.249023 17876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:22.250284 17876 hybrid_clock.cc:648] HybridClock initialized: now 1786142722250261 us; error 46 us; skew 500 ppm
Aug 07 22:45:22 dist-test-slave-rgb1 krb5kdc[17862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142722, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:22.258809 17876 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:45:22.259341 17876 webserver.cc:460] Webserver started at http://127.16.98.190:37469/ using document root <none> and password file <none>
I20260807 22:45:22.259708 17876 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:22.259784 17876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:22.259907 17876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:22.261258 17876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "7d585397cd91464db4d7fb87fdaa6ecf"
format_stamp: "Formatted at 2026-08-07 22:45:22 on dist-test-slave-rgb1"
server_key: "1ce92b721b270ea76cad53b1c3a032e4"
server_key_iv: "ba56508c5f75eee456087aadd180e544"
server_key_version: "encryptionkey@0"
I20260807 22:45:22.261617 17876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "7d585397cd91464db4d7fb87fdaa6ecf"
format_stamp: "Formatted at 2026-08-07 22:45:22 on dist-test-slave-rgb1"
server_key: "1ce92b721b270ea76cad53b1c3a032e4"
server_key_iv: "ba56508c5f75eee456087aadd180e544"
server_key_version: "encryptionkey@0"
I20260807 22:45:22.263168 17876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:45:22.264043 17891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:22.264250 17876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:22.264350 17876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "7d585397cd91464db4d7fb87fdaa6ecf"
format_stamp: "Formatted at 2026-08-07 22:45:22 on dist-test-slave-rgb1"
server_key: "1ce92b721b270ea76cad53b1c3a032e4"
server_key_iv: "ba56508c5f75eee456087aadd180e544"
server_key_version: "encryptionkey@0"
I20260807 22:45:22.264411 17876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:22.273542 17876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:22.275969 17876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:22.276100 17876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:22.282449 17876 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:36425
I20260807 22:45:22.282565 17955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:36425 every 8 connection(s)
I20260807 22:45:22.282938 17876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:22.283756 17956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:22.286255 17956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf: Bootstrap starting.
I20260807 22:45:22.286885 17956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:22.287196 17956 log.cc:824] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:22.288117 17956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf: No bootstrap required, opened a new log
I20260807 22:45:22.289299 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 17876
I20260807 22:45:22.289440 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:22.289655 16778 external_mini_cluster.cc:1428] Setting key 36c30158310d248d4687799be98a18ce
I20260807 22:45:22.291524 17956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d585397cd91464db4d7fb87fdaa6ecf" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36425 } }
I20260807 22:45:22.291738 17956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:22.291790 17956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d585397cd91464db4d7fb87fdaa6ecf, State: Initialized, Role: FOLLOWER
I20260807 22:45:22.292002 17956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d585397cd91464db4d7fb87fdaa6ecf" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36425 } }
I20260807 22:45:22.292099 17956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:22.292127 17956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:22.292184 17956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:22.293057 17956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d585397cd91464db4d7fb87fdaa6ecf" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36425 } }
I20260807 22:45:22.293202 17956 leader_election.cc:302] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d585397cd91464db4d7fb87fdaa6ecf; no voters:
I20260807 22:45:22.293417 17956 leader_election.cc:288] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:22.293531 17961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:22.293732 17961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [term 1 LEADER]: Becoming Leader. State: Replica: 7d585397cd91464db4d7fb87fdaa6ecf, State: Running, Role: LEADER
I20260807 22:45:22.293788 17956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:22.294427 17961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d585397cd91464db4d7fb87fdaa6ecf" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36425 } }
I20260807 22:45:22.295085 17963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d585397cd91464db4d7fb87fdaa6ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d585397cd91464db4d7fb87fdaa6ecf" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36425 } } }
I20260807 22:45:22.295176 17963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:22.295595 17962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d585397cd91464db4d7fb87fdaa6ecf. Latest consensus state: current_term: 1 leader_uuid: "7d585397cd91464db4d7fb87fdaa6ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d585397cd91464db4d7fb87fdaa6ecf" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36425 } } }
I20260807 22:45:22.295689 17962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:22.295853 17975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:22.296327 17975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:22.298054 17975 catalog_manager.cc:1347] Generated new cluster ID: 9e28163794ee457faa97874d76dc3f59
I20260807 22:45:22.298121 17975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 22:45:22 dist-test-slave-rgb1 krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:22.306442 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.290286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.16.98.190:36425)
0807 22:45:22.290558 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:22.290572 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:22.290623 (+ 51us) server_negotiation.cc:407] Connection header received
0807 22:45:22.291464 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:22.291474 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:22.291548 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:22.291736 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:22.292700 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.294432 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.295684 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.296294 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.296418 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.302921 (+ 6503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:22.302926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:22.302926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:22.302940 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:22.304781 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.305205 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.305209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.305209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.305304 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.305530 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.305531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.305531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.305783 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:22.305981 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:22.306084 (+ 103us) server_negotiation.cc:299] Negotiation successful
0807 22:45:22.306240 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":133,"threads_started":1}
I20260807 22:45:22.313691 17975 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:22.314455 17975 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:22.321216 17975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf: Generated new TSK 0
I20260807 22:45:22.321460 17975 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:45:22.361667 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36425
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:42877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:22.454865 17983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:22.455155 17983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:22.455186 17983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:22.456627 17983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:22.456691 17983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:22.456717 17983 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:45:22.456737 17983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:45:22.458793 17983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.98.190:36425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:22.459084 17983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:22.459298 17983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:22.464732 17988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:22.464726 17991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:22.464723 17989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:22.465147 17983 server_base.cc:1034] running on GCE node
I20260807 22:45:22.465390 17983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:22.465664 17983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:22.466976 17983 hybrid_clock.cc:648] HybridClock initialized: now 1786142722466964 us; error 75 us; skew 500 ppm
Aug 07 22:45:22 dist-test-slave-rgb1 krb5kdc[17862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142722, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:22.471153 17983 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:45:22.471817 17983 webserver.cc:460] Webserver started at http://127.16.98.129:40525/ using document root <none> and password file <none>
I20260807 22:45:22.472148 17983 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:22.472234 17983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:22.472402 17983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:22.474015 17983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "5d77444215074c87a27e8f76207a4d28"
format_stamp: "Formatted at 2026-08-07 22:45:22 on dist-test-slave-rgb1"
server_key: "4587af913da6f929624438be1fbadb59"
server_key_iv: "f239734c9b0517ca79d168c67947c26a"
server_key_version: "encryptionkey@0"
I20260807 22:45:22.474483 17983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "5d77444215074c87a27e8f76207a4d28"
format_stamp: "Formatted at 2026-08-07 22:45:22 on dist-test-slave-rgb1"
server_key: "4587af913da6f929624438be1fbadb59"
server_key_iv: "f239734c9b0517ca79d168c67947c26a"
server_key_version: "encryptionkey@0"
I20260807 22:45:22.476593 17983 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 22:45:22.477626 17998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:22.477849 17983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:22.477957 17983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "5d77444215074c87a27e8f76207a4d28"
format_stamp: "Formatted at 2026-08-07 22:45:22 on dist-test-slave-rgb1"
server_key: "4587af913da6f929624438be1fbadb59"
server_key_iv: "f239734c9b0517ca79d168c67947c26a"
server_key_version: "encryptionkey@0"
I20260807 22:45:22.478068 17983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:22.516225 17983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:22.519647 17983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:22.519840 17983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:22.520529 17983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:22.520620 17983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:22.520674 17983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:22.520706 17983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:22.528805 17983 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:39687
I20260807 22:45:22.529045 18117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:39687 every 8 connection(s)
I20260807 22:45:22.529255 17983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
Aug 07 22:45:22 dist-test-slave-rgb1 krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142722, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:22.533362 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.521970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:35621 (local address 127.16.98.190:36425)
0807 22:45:22.522036 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:22.522040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:22.522052 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:45:22.522873 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:22.522875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:22.522889 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:22.522965 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:22.524049 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.524877 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.526628 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.526956 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.527052 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.530230 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:22.530232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:22.530232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:22.530242 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:22.531606 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.532100 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.532102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.532103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.532172 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.532464 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.532465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.532466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.532691 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:22.532777 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:22.533157 (+ 380us) server_negotiation.cc:299] Negotiation successful
0807 22:45:22.533257 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 22:45:22.533586 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.521600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36425 (local address 127.16.98.129:35621)
0807 22:45:22.521812 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:22.521872 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:22.522736 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:22.523135 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:22.523141 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:22.523587 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:22.523908 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.523917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.524981 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.524982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.526513 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.526517 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.527077 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.527079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.527245 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.527965 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:22.527977 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:22.530105 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:22.531727 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:22.531736 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:22.531739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:22.531994 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:22.532259 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:22.532260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:22.532261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:22.532383 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:22.532783 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:22.532791 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:22.532947 (+ 156us) client_negotiation.cc:769] Sending connection context
0807 22:45:22.533132 (+ 185us) client_negotiation.cc:240] Negotiation successful
0807 22:45:22.533322 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":100,"threads_started":1}
I20260807 22:45:22.534277 18118 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36425
I20260807 22:45:22.534380 18118 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:22.534598 18118 heartbeater.cc:511] Master 127.16.98.190:36425 requested a full tablet report, sending...
I20260807 22:45:22.535317 17908 ts_manager.cc:194] Registered new tserver with Master: 5d77444215074c87a27e8f76207a4d28 (127.16.98.129:39687)
I20260807 22:45:22.536300 17908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:35621
I20260807 22:45:22.539186 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 17983
I20260807 22:45:22.539341 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:45:22.539629 16778 external_mini_cluster.cc:1428] Setting key 6fad85bb178cd303486e12943590f173
I20260807 22:45:22.544111 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.535496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36425 (local address 127.16.98.129:49429)
0807 22:45:22.535659 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:22.535671 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:22.535747 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:22.535888 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:22.535889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:22.536110 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:22.536173 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.536175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.537310 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.537311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.537994 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.537997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.538604 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.538606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.538681 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.539319 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:22.539326 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:22.539780 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:22.541995 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:22.542007 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:22.542008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:22.542611 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:22.543127 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:22.543129 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:22.543129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:22.543249 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:22.543616 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:22.543617 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:22.543731 (+ 114us) client_negotiation.cc:769] Sending connection context
0807 22:45:22.543828 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 22:45:22.544012 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260807 22:45:22.544293 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.535601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:49429 (local address 127.16.98.190:36425)
0807 22:45:22.535648 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:22.535651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:22.535775 (+ 124us) server_negotiation.cc:407] Connection header received
0807 22:45:22.535801 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:22.535802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:22.535812 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:22.535881 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:22.536273 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.537186 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.538160 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.538419 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.538538 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.539915 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:22.539918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:22.539919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:22.539929 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:22.541791 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.542873 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.542874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.542875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.542968 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.543353 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.543354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.543355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.543520 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:22.543606 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:22.543900 (+ 294us) server_negotiation.cc:299] Negotiation successful
0807 22:45:22.544148 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:45:22.580655 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36425
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:42877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:22.670660 18128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:22.670920 18128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:22.670964 18128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:22.672169 18128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:22.672237 18128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:22.672274 18128 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:45:22.672333 18128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:45:22.674093 18128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.98.190:36425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:22.674721 18128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:22.675050 18128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:22.680936 18134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:22.680936 18136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:22.680956 18128 server_base.cc:1034] running on GCE node
W20260807 22:45:22.680933 18133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:22.681469 18128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:22.681792 18128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:22.683029 18128 hybrid_clock.cc:648] HybridClock initialized: now 1786142722682964 us; error 79 us; skew 500 ppm
Aug 07 22:45:22 dist-test-slave-rgb1 krb5kdc[17862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142722, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:22.686151 18128 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:45:22.686691 18128 webserver.cc:460] Webserver started at http://127.16.98.130:35019/ using document root <none> and password file <none>
I20260807 22:45:22.686954 18128 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:22.687032 18128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:22.687194 18128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:22.688337 18128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1"
format_stamp: "Formatted at 2026-08-07 22:45:22 on dist-test-slave-rgb1"
server_key: "cb5691f845f442d15832a5212c617978"
server_key_iv: "6b198dd37f03c45f4e61472676230205"
server_key_version: "encryptionkey@0"
I20260807 22:45:22.688767 18128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1"
format_stamp: "Formatted at 2026-08-07 22:45:22 on dist-test-slave-rgb1"
server_key: "cb5691f845f442d15832a5212c617978"
server_key_iv: "6b198dd37f03c45f4e61472676230205"
server_key_version: "encryptionkey@0"
I20260807 22:45:22.690485 18128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:22.691632 18143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:22.691964 18128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:22.692085 18128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1"
format_stamp: "Formatted at 2026-08-07 22:45:22 on dist-test-slave-rgb1"
server_key: "cb5691f845f442d15832a5212c617978"
server_key_iv: "6b198dd37f03c45f4e61472676230205"
server_key_version: "encryptionkey@0"
I20260807 22:45:22.692194 18128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:22.701781 18128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:22.704314 18128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:22.704504 18128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:22.705122 18128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:22.705186 18128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:22.705227 18128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:22.705276 18128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:22.712716 18128 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:40377
I20260807 22:45:22.712813 18261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:40377 every 8 connection(s)
I20260807 22:45:22.713179 18128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
Aug 07 22:45:22 dist-test-slave-rgb1 krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142722, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:22.717568 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 18128
I20260807 22:45:22.717671 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:45:22.718011 16778 external_mini_cluster.cc:1428] Setting key e17cbbd26fde68fb72188f0b064b5352
I20260807 22:45:22.718235 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.705941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36425 (local address 127.16.98.130:54571)
0807 22:45:22.706236 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:22.706282 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:22.707021 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:22.707414 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:22.707419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:22.707847 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:22.708181 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.708189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.709376 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.709378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.710869 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.710872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.711416 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.711417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.711609 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.712326 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:22.712337 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:22.714439 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:22.716457 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:22.716467 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:22.716470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:22.716811 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:22.717045 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:22.717046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:22.717046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:22.717176 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:22.717477 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:22.717486 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:22.717665 (+ 179us) client_negotiation.cc:769] Sending connection context
0807 22:45:22.717784 (+ 119us) client_negotiation.cc:240] Negotiation successful
0807 22:45:22.717952 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":94,"threads_started":1}
I20260807 22:45:22.718526 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.705997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:54571 (local address 127.16.98.190:36425)
0807 22:45:22.706138 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:22.706140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:22.706370 (+ 230us) server_negotiation.cc:407] Connection header received
0807 22:45:22.707191 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:22.707193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:22.707206 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:22.707280 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:22.708354 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.709252 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.711009 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.711272 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.711401 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.714602 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:22.714604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:22.714604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:22.714614 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:22.716343 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.716897 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.716898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.716899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.716971 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.717259 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.717260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.717260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.717392 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:22.717474 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:22.718276 (+ 802us) server_negotiation.cc:299] Negotiation successful
0807 22:45:22.718442 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260807 22:45:22.718958 18262 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36425
I20260807 22:45:22.719072 18262 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:22.719277 18262 heartbeater.cc:511] Master 127.16.98.190:36425 requested a full tablet report, sending...
I20260807 22:45:22.719683 17908 ts_manager.cc:194] Registered new tserver with Master: c432cc2bc1fb4e55b11b9dc325b9e3f1 (127.16.98.130:40377)
I20260807 22:45:22.720191 17908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:54571
I20260807 22:45:22.726619 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.720357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:59475 (local address 127.16.98.190:36425)
0807 22:45:22.720408 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:22.720411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:22.720531 (+ 120us) server_negotiation.cc:407] Connection header received
0807 22:45:22.720750 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:22.720751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:22.720764 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:22.720828 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:22.721198 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.721679 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.722402 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.722604 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.722683 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.723799 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:22.723800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:22.723801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:22.723808 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:22.725317 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.725707 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.725708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.725708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.725772 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.725987 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.725989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.725989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.726162 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:22.726251 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:22.726426 (+ 175us) server_negotiation.cc:299] Negotiation successful
0807 22:45:22.726514 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260807 22:45:22.726624 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.720317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36425 (local address 127.16.98.130:59475)
0807 22:45:22.720472 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:22.720483 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:22.720544 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:22.720873 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:22.720875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:22.721059 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:22.721112 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.721114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.721784 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.721785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.722301 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.722302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.722682 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.722683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.722764 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.723303 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:22.723308 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:22.723700 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:22.725438 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:22.725440 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:22.725440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:22.725629 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:22.725848 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:22.725849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:22.725849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:22.725919 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:22.726242 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:22.726243 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:22.726298 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 22:45:22.726382 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 22:45:22.726518 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:45:22.753260 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36425
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:42877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:22.843465 18272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:22.843758 18272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:22.843809 18272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:22.845791 18272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:22.845881 18272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:22.845927 18272 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:45:22.845970 18272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:45:22.847759 18272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.98.190:36425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:22.848093 18272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:22.848392 18272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:22.854837 18278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:22.854696 18277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:22.855012 18272 server_base.cc:1034] running on GCE node
W20260807 22:45:22.854779 18280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:22.855490 18272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:22.855733 18272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:22.856895 18272 hybrid_clock.cc:648] HybridClock initialized: now 1786142722856838 us; error 81 us; skew 500 ppm
Aug 07 22:45:22 dist-test-slave-rgb1 krb5kdc[17862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142722, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:22.859766 18272 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:45:22.860365 18272 webserver.cc:460] Webserver started at http://127.16.98.131:34841/ using document root <none> and password file <none>
I20260807 22:45:22.860664 18272 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:22.860723 18272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:22.860869 18272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:22.862195 18272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "23fec5f96113472d8e22766fa6fa8d00"
format_stamp: "Formatted at 2026-08-07 22:45:22 on dist-test-slave-rgb1"
server_key: "5e6c442a3d83f24dd260a19013613760"
server_key_iv: "dd36073d6bda79841f06cdad77b54aa3"
server_key_version: "encryptionkey@0"
I20260807 22:45:22.862583 18272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "23fec5f96113472d8e22766fa6fa8d00"
format_stamp: "Formatted at 2026-08-07 22:45:22 on dist-test-slave-rgb1"
server_key: "5e6c442a3d83f24dd260a19013613760"
server_key_iv: "dd36073d6bda79841f06cdad77b54aa3"
server_key_version: "encryptionkey@0"
I20260807 22:45:22.864326 18272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:45:22.865435 18288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:22.865813 18272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:22.865938 18272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "23fec5f96113472d8e22766fa6fa8d00"
format_stamp: "Formatted at 2026-08-07 22:45:22 on dist-test-slave-rgb1"
server_key: "5e6c442a3d83f24dd260a19013613760"
server_key_iv: "dd36073d6bda79841f06cdad77b54aa3"
server_key_version: "encryptionkey@0"
I20260807 22:45:22.866026 18272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:22.886062 18272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:22.889398 18272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:22.889632 18272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:22.890208 18272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:22.890271 18272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:22.890307 18272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:22.890359 18272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:22.898129 18272 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:38101
I20260807 22:45:22.898212 18410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:38101 every 8 connection(s)
Aug 07 22:45:22 dist-test-slave-rgb1 krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142722, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:22.898548 18272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:45:22.900223 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 18272
I20260807 22:45:22.900324 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:45:22.900534 16778 external_mini_cluster.cc:1428] Setting key 74466e0017a9d867f84a8bba394b1d4a
I20260807 22:45:22.903383 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.891002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:39187 (local address 127.16.98.190:36425)
0807 22:45:22.891058 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:22.891060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:22.891204 (+ 144us) server_negotiation.cc:407] Connection header received
0807 22:45:22.891812 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:22.891814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:22.891829 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:22.891905 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:22.893111 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.893968 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.895708 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.896000 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.896070 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.899172 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:22.899174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:22.899174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:22.899181 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:22.901345 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.902155 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.902157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.902157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.902233 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.902584 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.902585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.902585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.902718 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:22.902787 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:22.903223 (+ 436us) server_negotiation.cc:299] Negotiation successful
0807 22:45:22.903306 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260807 22:45:22.903554 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.890874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36425 (local address 127.16.98.131:39187)
0807 22:45:22.891062 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:22.891117 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:22.891643 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:22.892109 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:22.892114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:22.892613 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:22.892952 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.892961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.894109 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.894111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.895609 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.895613 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.896108 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.896109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.896278 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.896982 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:22.896994 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:22.899079 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:22.901540 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:22.901550 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:22.901553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:22.902042 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:22.902333 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:22.902334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:22.902334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:22.902459 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:22.902855 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:22.902862 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:22.903063 (+ 201us) client_negotiation.cc:769] Sending connection context
0807 22:45:22.903170 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 22:45:22.903312 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260807 22:45:22.904095 18411 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36425
I20260807 22:45:22.904187 18411 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:22.904390 18411 heartbeater.cc:511] Master 127.16.98.190:36425 requested a full tablet report, sending...
I20260807 22:45:22.904896 17907 ts_manager.cc:194] Registered new tserver with Master: 23fec5f96113472d8e22766fa6fa8d00 (127.16.98.131:38101)
I20260807 22:45:22.905431 17907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:39187
I20260807 22:45:22.912525 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.905266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36425 (local address 127.16.98.131:40475)
0807 22:45:22.905429 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:22.905442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:22.905537 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:22.905923 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:22.905923 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:22.906111 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:22.906175 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.906177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.906910 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.906911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.907487 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.907490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.907864 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.907865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.907932 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.908622 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:22.908628 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:22.908959 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:22.910775 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:22.910778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:22.910778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:22.911069 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:22.911507 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:22.911508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:22.911509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:22.911607 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:22.912094 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:22.912095 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:22.912160 (+ 65us) client_negotiation.cc:769] Sending connection context
0807 22:45:22.912255 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 22:45:22.912392 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260807 22:45:22.912657 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.905383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:40475 (local address 127.16.98.190:36425)
0807 22:45:22.905432 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:22.905434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:22.905518 (+ 84us) server_negotiation.cc:407] Connection header received
0807 22:45:22.905757 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:22.905757 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:22.905769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:22.905875 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:22.906264 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.906784 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.907586 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.907771 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.907836 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.909065 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:22.909066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:22.909066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:22.909073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:22.910619 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.911272 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.911275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.911275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.911365 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.911756 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.911758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.911758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.911930 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:22.912049 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:22.912359 (+ 310us) server_negotiation.cc:299] Negotiation successful
0807 22:45:22.912513 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 22:45:22.913214 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 22:45:22 dist-test-slave-rgb1 krb5kdc[17862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142722, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 22:45:22 dist-test-slave-rgb1 krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142722, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:22.940524 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.931616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.16.98.190:36425)
0807 22:45:22.931686 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:22.931688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:22.931697 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:45:22.931723 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:22.931723 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:22.931734 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:22.931799 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:22.932529 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.933271 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.934284 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.934480 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.934546 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.937506 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:22.937508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:22.937509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:22.937518 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:22.939210 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.939640 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.939642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.939642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.939706 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:22.939924 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:22.939924 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:22.939925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:22.940060 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:22.940182 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:22.940324 (+ 142us) server_negotiation.cc:299] Negotiation successful
0807 22:45:22.940413 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 22:45:22.942513 17907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:45:22.944195 17907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:22.952086 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.947314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.16.98.131:38101)
0807 22:45:22.947496 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:22.947501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:22.947725 (+ 224us) server_negotiation.cc:407] Connection header received
0807 22:45:22.947873 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:22.947876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:22.947895 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:22.947961 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:22.948253 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.949180 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.950849 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.951583 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.951664 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.951715 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:22.951917 (+ 202us) server_negotiation.cc:299] Negotiation successful
0807 22:45:22.951985 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":180,"threads_started":1}
I20260807 22:45:22.952075 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.947210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:38101 (local address 127.0.0.1:55260)
0807 22:45:22.947618 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:22.947631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:22.947763 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:22.948068 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:22.948069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:22.948071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:22.948148 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.948150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.949287 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.949288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.950757 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.950760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.951696 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.951697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.951780 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.951799 (+ 19us) client_negotiation.cc:769] Sending connection context
0807 22:45:22.951890 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 22:45:22.951970 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":3,"thread_start_us":98,"threads_started":1}
I20260807 22:45:22.952716 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.947769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.16.98.130:40377)
0807 22:45:22.948003 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:22.948008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:22.948169 (+ 161us) server_negotiation.cc:407] Connection header received
0807 22:45:22.948221 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:22.948224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:22.948240 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:22.948318 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:22.948514 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.949644 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.950756 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.951538 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.951632 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.951704 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:22.952516 (+ 812us) server_negotiation.cc:299] Negotiation successful
0807 22:45:22.952596 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":99,"threads_started":1}
I20260807 22:45:22.952965 18342 tablet_service.cc:1467] Processing CreateTablet for tablet 9b61454a39b34956aeab7a41f801e79d (DEFAULT_TABLE table=test-table [id=aca9b4d2ab5047db9d667ebfdf16bf15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:22.953210 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.947204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:39687 (local address 127.0.0.1:60558)
0807 22:45:22.947581 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:22.947629 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:22.947729 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:22.948048 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:22.948051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:22.948053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:22.948146 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.948149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.949598 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.949599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.950768 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.950769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.952974 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.952975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.953041 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.953044 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:45:22.953089 (+ 45us) client_negotiation.cc:240] Negotiation successful
0807 22:45:22.953123 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":76,"threads_started":1}
I20260807 22:45:22.953292 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.947315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.16.98.129:39687)
0807 22:45:22.947528 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:22.947532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:22.947826 (+ 294us) server_negotiation.cc:407] Connection header received
0807 22:45:22.947879 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:22.947881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:22.947898 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:22.947970 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:22.948240 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.949503 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.951753 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.952807 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.952887 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.952943 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:22.953136 (+ 193us) server_negotiation.cc:299] Negotiation successful
0807 22:45:22.953188 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":105,"threads_started":1}
I20260807 22:45:22.953394 18342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b61454a39b34956aeab7a41f801e79d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:22.953964 18035 tablet_service.cc:1467] Processing CreateTablet for tablet 9b61454a39b34956aeab7a41f801e79d (DEFAULT_TABLE table=test-table [id=aca9b4d2ab5047db9d667ebfdf16bf15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:22.954324 18035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b61454a39b34956aeab7a41f801e79d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:22.956655 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:22.947438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:40377 (local address 127.0.0.1:57470)
0807 22:45:22.948072 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:22.948083 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:22.948145 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:22.948387 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:22.948388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:22.948391 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:22.948434 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.948435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.949854 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.949854 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.950642 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:22.950645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:22.951683 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:22.951684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:22.951780 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:22.951799 (+ 19us) client_negotiation.cc:769] Sending connection context
0807 22:45:22.951890 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 22:45:22.951978 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":48,"thread_start_us":129,"threads_started":1}
I20260807 22:45:22.957144 18434 tablet_bootstrap.cc:492] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00: Bootstrap starting.
I20260807 22:45:22.957518 18194 tablet_service.cc:1467] Processing CreateTablet for tablet 9b61454a39b34956aeab7a41f801e79d (DEFAULT_TABLE table=test-table [id=aca9b4d2ab5047db9d667ebfdf16bf15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:22.957906 18194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b61454a39b34956aeab7a41f801e79d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:22.958256 18435 tablet_bootstrap.cc:492] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28: Bootstrap starting.
I20260807 22:45:22.958187 18434 tablet_bootstrap.cc:654] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:22.958706 18434 log.cc:824] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:22.959173 18435 tablet_bootstrap.cc:654] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:22.959504 18435 log.cc:824] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:22.960355 18435 tablet_bootstrap.cc:492] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28: No bootstrap required, opened a new log
I20260807 22:45:22.960507 18435 ts_tablet_manager.cc:1397] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 22:45:22.961526 18438 tablet_bootstrap.cc:492] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1: Bootstrap starting.
I20260807 22:45:22.962162 18434 tablet_bootstrap.cc:492] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00: No bootstrap required, opened a new log
I20260807 22:45:22.962225 18434 ts_tablet_manager.cc:1397] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260807 22:45:22.962507 18438 tablet_bootstrap.cc:654] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:22.963326 18438 log.cc:824] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:22.963636 18435 raft_consensus.cc:348] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } }
I20260807 22:45:22.963853 18435 raft_consensus.cc:374] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:22.963905 18435 raft_consensus.cc:729] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d77444215074c87a27e8f76207a4d28, State: Initialized, Role: FOLLOWER
I20260807 22:45:22.964061 18435 consensus_queue.cc:260] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } }
I20260807 22:45:22.964210 18438 tablet_bootstrap.cc:492] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1: No bootstrap required, opened a new log
I20260807 22:45:22.964290 18438 ts_tablet_manager.cc:1397] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:45:22.964375 18435 ts_tablet_manager.cc:1428] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260807 22:45:22.964448 18118 heartbeater.cc:503] Master 127.16.98.190:36425 was elected leader, sending a full tablet report...
I20260807 22:45:22.964360 18434 raft_consensus.cc:348] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } }
I20260807 22:45:22.964510 18434 raft_consensus.cc:374] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:22.964586 18434 raft_consensus.cc:729] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23fec5f96113472d8e22766fa6fa8d00, State: Initialized, Role: FOLLOWER
I20260807 22:45:22.964694 18434 consensus_queue.cc:260] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } }
I20260807 22:45:22.964972 18434 ts_tablet_manager.cc:1428] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 22:45:22.965024 18411 heartbeater.cc:503] Master 127.16.98.190:36425 was elected leader, sending a full tablet report...
I20260807 22:45:22.966408 18438 raft_consensus.cc:348] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } }
I20260807 22:45:22.966583 18438 raft_consensus.cc:374] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:22.966609 18438 raft_consensus.cc:729] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c432cc2bc1fb4e55b11b9dc325b9e3f1, State: Initialized, Role: FOLLOWER
I20260807 22:45:22.966711 18438 consensus_queue.cc:260] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } }
I20260807 22:45:22.966976 18438 ts_tablet_manager.cc:1428] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:45:22.967134 18262 heartbeater.cc:503] Master 127.16.98.190:36425 was elected leader, sending a full tablet report...
W20260807 22:45:23.030133 18119 tablet.cc:2369] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:45:23.036371 18442 raft_consensus.cc:482] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:23.036537 18442 raft_consensus.cc:504] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } }
I20260807 22:45:23.037009 18442 leader_election.cc:288] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23fec5f96113472d8e22766fa6fa8d00 (127.16.98.131:38101), 5d77444215074c87a27e8f76207a4d28 (127.16.98.129:39687)
I20260807 22:45:23.041416 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.037284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:39687 (local address 127.16.98.130:44459)
0807 22:45:23.037568 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:23.037600 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:23.037657 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:23.037787 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:23.037788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:23.037790 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:23.037860 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.037861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.038773 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.038774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.040182 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.040186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.041077 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.041079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.041188 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.041192 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:23.041278 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 22:45:23.041315 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":151,"thread_start_us":177,"threads_started":1}
I20260807 22:45:23.041453 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.037337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:44459 (local address 127.16.98.129:39687)
0807 22:45:23.037390 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:23.037392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:23.037648 (+ 256us) server_negotiation.cc:407] Connection header received
0807 22:45:23.037718 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:23.037718 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:23.037726 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:23.037767 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:23.037986 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.038660 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.040316 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.040944 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.041022 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.041035 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:23.041309 (+ 274us) server_negotiation.cc:299] Negotiation successful
0807 22:45:23.041375 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260807 22:45:23.041497 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.037431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:55413 (local address 127.16.98.131:38101)
0807 22:45:23.037487 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:23.037489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:23.037502 (+ 13us) server_negotiation.cc:407] Connection header received
0807 22:45:23.037528 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:23.037528 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:23.037536 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:23.037600 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:23.037828 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.038608 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.040316 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.040898 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.041053 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.041064 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:23.041362 (+ 298us) server_negotiation.cc:299] Negotiation successful
0807 22:45:23.041418 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260807 22:45:23.041708 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.037284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:38101 (local address 127.16.98.130:55413)
0807 22:45:23.037331 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:23.037344 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:23.037401 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:23.037633 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:23.037634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:23.037636 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:23.037716 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.037719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.038773 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.038780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.040199 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.040201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.041092 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.041093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.041195 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.041197 (+ 2us) client_negotiation.cc:769] Sending connection context
0807 22:45:23.041245 (+ 48us) client_negotiation.cc:240] Negotiation successful
0807 22:45:23.041301 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":86}
I20260807 22:45:23.042078 18055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b61454a39b34956aeab7a41f801e79d" candidate_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d77444215074c87a27e8f76207a4d28" is_pre_election: true
I20260807 22:45:23.042078 18362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b61454a39b34956aeab7a41f801e79d" candidate_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23fec5f96113472d8e22766fa6fa8d00" is_pre_election: true
I20260807 22:45:23.042258 18055 raft_consensus.cc:2454] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c432cc2bc1fb4e55b11b9dc325b9e3f1 in term 0.
I20260807 22:45:23.042266 18362 raft_consensus.cc:2454] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c432cc2bc1fb4e55b11b9dc325b9e3f1 in term 0.
I20260807 22:45:23.042563 18145 leader_election.cc:302] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d77444215074c87a27e8f76207a4d28, c432cc2bc1fb4e55b11b9dc325b9e3f1; no voters:
I20260807 22:45:23.042719 18442 raft_consensus.cc:2781] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:23.042801 18442 raft_consensus.cc:482] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:23.042840 18442 raft_consensus.cc:3037] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:23.043895 18442 raft_consensus.cc:504] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } }
I20260807 22:45:23.044072 18442 leader_election.cc:288] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [CANDIDATE]: Term 1 election: Requested vote from peers 23fec5f96113472d8e22766fa6fa8d00 (127.16.98.131:38101), 5d77444215074c87a27e8f76207a4d28 (127.16.98.129:39687)
I20260807 22:45:23.044245 18055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b61454a39b34956aeab7a41f801e79d" candidate_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d77444215074c87a27e8f76207a4d28"
I20260807 22:45:23.044239 18362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b61454a39b34956aeab7a41f801e79d" candidate_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23fec5f96113472d8e22766fa6fa8d00"
I20260807 22:45:23.044348 18362 raft_consensus.cc:3037] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:23.044351 18055 raft_consensus.cc:3037] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:23.045292 18055 raft_consensus.cc:2454] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c432cc2bc1fb4e55b11b9dc325b9e3f1 in term 1.
I20260807 22:45:23.045406 18362 raft_consensus.cc:2454] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c432cc2bc1fb4e55b11b9dc325b9e3f1 in term 1.
I20260807 22:45:23.045548 18145 leader_election.cc:302] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d77444215074c87a27e8f76207a4d28, c432cc2bc1fb4e55b11b9dc325b9e3f1; no voters:
I20260807 22:45:23.045651 18442 raft_consensus.cc:2781] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:23.045835 18442 raft_consensus.cc:686] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 1 LEADER]: Becoming Leader. State: Replica: c432cc2bc1fb4e55b11b9dc325b9e3f1, State: Running, Role: LEADER
I20260807 22:45:23.045969 18442 consensus_queue.cc:237] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } }
I20260807 22:45:23.047078 17908 catalog_manager.cc:5495] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c432cc2bc1fb4e55b11b9dc325b9e3f1 (127.16.98.130). New cstate: current_term: 1 leader_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } health_report { overall_health: UNKNOWN } } }
I20260807 22:45:23.071265 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.068175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.16.98.130:40377)
0807 22:45:23.068260 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:23.068262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:23.068273 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:45:23.068388 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:23.068389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:23.068414 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:23.068488 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:23.068758 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.069284 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.070211 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.070440 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.070507 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.070779 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:23.070889 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:23.071002 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:23.071096 (+ 94us) server_negotiation.cc:299] Negotiation successful
0807 22:45:23.071120 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 22:45:23.073628 18362 raft_consensus.cc:1261] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00 [term 1 FOLLOWER]: Refusing update from remote peer c432cc2bc1fb4e55b11b9dc325b9e3f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:23.073629 18055 raft_consensus.cc:1261] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28 [term 1 FOLLOWER]: Refusing update from remote peer c432cc2bc1fb4e55b11b9dc325b9e3f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:23.074064 18442 consensus_queue.cc:1035] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:23.074297 18444 consensus_queue.cc:1035] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:23.081300 17908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:23.081419 17908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:23.082324 17908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf: Sending DeleteTablet for 3 replicas of tablet 9b61454a39b34956aeab7a41f801e79d
I20260807 22:45:23.082876 18342 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b61454a39b34956aeab7a41f801e79d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:23 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:55260
I20260807 22:45:23.082952 18035 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b61454a39b34956aeab7a41f801e79d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:23 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:60558
I20260807 22:45:23.083156 18194 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b61454a39b34956aeab7a41f801e79d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:23 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:57470
I20260807 22:45:23.083280 18457 tablet_replica.cc:330] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00: stopping tablet replica
I20260807 22:45:23.083407 18459 tablet_replica.cc:330] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1: stopping tablet replica
I20260807 22:45:23.083458 18457 raft_consensus.cc:2229] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:23.083555 18459 raft_consensus.cc:2229] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:45:23.083665 18459 raft_consensus.cc:2258] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:23.083640 17908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:45:23.083807 17908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:23.084005 18457 raft_consensus.cc:2258] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:23.084594 18459 ts_tablet_manager.cc:1905] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:23.085526 18035 tablet_service.cc:1467] Processing CreateTablet for tablet e5b404142abf4ce4b8e674e28cbe4010 (DEFAULT_TABLE table=test-table [id=8a960f7b57354b3ea0b3cbc7127bdedf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:23.085548 18342 tablet_service.cc:1467] Processing CreateTablet for tablet e5b404142abf4ce4b8e674e28cbe4010 (DEFAULT_TABLE table=test-table [id=8a960f7b57354b3ea0b3cbc7127bdedf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:23.085660 18035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b404142abf4ce4b8e674e28cbe4010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:23.085660 18342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b404142abf4ce4b8e674e28cbe4010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:23.087034 18435 tablet_bootstrap.cc:492] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28: Bootstrap starting.
I20260807 22:45:23.087446 18435 tablet_bootstrap.cc:654] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:23.087579 18194 tablet_service.cc:1467] Processing CreateTablet for tablet e5b404142abf4ce4b8e674e28cbe4010 (DEFAULT_TABLE table=test-table [id=8a960f7b57354b3ea0b3cbc7127bdedf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:23.087718 18194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b404142abf4ce4b8e674e28cbe4010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:23.087821 18457 ts_tablet_manager.cc:1905] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:23.088686 18434 tablet_bootstrap.cc:492] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00: Bootstrap starting.
I20260807 22:45:23.089146 18434 tablet_bootstrap.cc:654] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:23.089309 18438 tablet_bootstrap.cc:492] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1: Bootstrap starting.
I20260807 22:45:23.089730 18438 tablet_bootstrap.cc:654] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:23.090108 18457 ts_tablet_manager.cc:1918] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:23.090324 18457 log.cc:1196] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/9b61454a39b34956aeab7a41f801e79d
I20260807 22:45:23.090523 18434 tablet_bootstrap.cc:492] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00: No bootstrap required, opened a new log
I20260807 22:45:23.090579 18434 ts_tablet_manager.cc:1397] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260807 22:45:23.090739 18458 tablet_replica.cc:330] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28: stopping tablet replica
I20260807 22:45:23.090750 18434 raft_consensus.cc:348] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.090876 18458 raft_consensus.cc:2229] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:23.090857 18434 raft_consensus.cc:374] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:23.090955 18434 raft_consensus.cc:729] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23fec5f96113472d8e22766fa6fa8d00, State: Initialized, Role: FOLLOWER
I20260807 22:45:23.090992 18459 ts_tablet_manager.cc:1918] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:23.091015 18458 raft_consensus.cc:2258] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:23.091008 18434 consensus_queue.cc:260] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.091059 18459 log.cc:1196] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/9b61454a39b34956aeab7a41f801e79d
I20260807 22:45:23.091115 18434 ts_tablet_manager.cc:1428] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260807 22:45:23.091329 18459 ts_tablet_manager.cc:1939] T 9b61454a39b34956aeab7a41f801e79d P c432cc2bc1fb4e55b11b9dc325b9e3f1: Deleting consensus metadata
I20260807 22:45:23.091454 18435 tablet_bootstrap.cc:492] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28: No bootstrap required, opened a new log
I20260807 22:45:23.091337 18438 tablet_bootstrap.cc:492] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1: No bootstrap required, opened a new log
I20260807 22:45:23.091500 18435 ts_tablet_manager.cc:1397] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260807 22:45:23.091507 18438 ts_tablet_manager.cc:1397] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 22:45:23.091661 18435 raft_consensus.cc:348] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.091703 18438 raft_consensus.cc:348] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.091773 18438 raft_consensus.cc:374] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:23.091741 18435 raft_consensus.cc:374] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:23.091794 18438 raft_consensus.cc:729] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c432cc2bc1fb4e55b11b9dc325b9e3f1, State: Initialized, Role: FOLLOWER
I20260807 22:45:23.091800 18435 raft_consensus.cc:729] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d77444215074c87a27e8f76207a4d28, State: Initialized, Role: FOLLOWER
I20260807 22:45:23.091846 18438 consensus_queue.cc:260] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.091851 18435 consensus_queue.cc:260] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.091940 18438 ts_tablet_manager.cc:1428] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:23.091996 18435 ts_tablet_manager.cc:1428] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260807 22:45:23.091734 18458 ts_tablet_manager.cc:1905] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:23.092325 17894 catalog_manager.cc:4841] TS c432cc2bc1fb4e55b11b9dc325b9e3f1 (127.16.98.130:40377): tablet 9b61454a39b34956aeab7a41f801e79d (table test-table [id=aca9b4d2ab5047db9d667ebfdf16bf15]) successfully deleted
I20260807 22:45:23.090848 18457 ts_tablet_manager.cc:1939] T 9b61454a39b34956aeab7a41f801e79d P 23fec5f96113472d8e22766fa6fa8d00: Deleting consensus metadata
I20260807 22:45:23.093181 17894 catalog_manager.cc:4841] TS 23fec5f96113472d8e22766fa6fa8d00 (127.16.98.131:38101): tablet 9b61454a39b34956aeab7a41f801e79d (table test-table [id=aca9b4d2ab5047db9d667ebfdf16bf15]) successfully deleted
I20260807 22:45:23.094172 18458 ts_tablet_manager.cc:1918] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:23.094261 18458 log.cc:1196] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/wal/wals/9b61454a39b34956aeab7a41f801e79d
I20260807 22:45:23.095044 18458 ts_tablet_manager.cc:1939] T 9b61454a39b34956aeab7a41f801e79d P 5d77444215074c87a27e8f76207a4d28: Deleting consensus metadata
I20260807 22:45:23.095489 17893 catalog_manager.cc:4841] TS 5d77444215074c87a27e8f76207a4d28 (127.16.98.129:39687): tablet 9b61454a39b34956aeab7a41f801e79d (table test-table [id=aca9b4d2ab5047db9d667ebfdf16bf15]) successfully deleted
W20260807 22:45:23.149168 18412 tablet.cc:2369] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:45:23.177474 18440 raft_consensus.cc:482] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:23.177660 18440 raft_consensus.cc:504] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.178046 18440 leader_election.cc:288] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c432cc2bc1fb4e55b11b9dc325b9e3f1 (127.16.98.130:40377), 23fec5f96113472d8e22766fa6fa8d00 (127.16.98.131:38101)
I20260807 22:45:23.181936 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.178613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:50439 (local address 127.16.98.131:38101)
0807 22:45:23.178662 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:23.178665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:23.178719 (+ 54us) server_negotiation.cc:407] Connection header received
0807 22:45:23.178751 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:23.178752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:23.178764 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:23.178826 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:23.178972 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.179802 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.181204 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.181573 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.181663 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.181673 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:23.181825 (+ 152us) server_negotiation.cc:299] Negotiation successful
0807 22:45:23.181863 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 22:45:23.181955 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.178476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:38101 (local address 127.16.98.129:50439)
0807 22:45:23.178607 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:23.178616 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:23.178659 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:23.178839 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:23.178839 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:23.178840 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:23.178897 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.178898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.179901 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.179902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.181076 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.181078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.181663 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.181665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.181747 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.181752 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:45:23.181817 (+ 65us) client_negotiation.cc:240] Negotiation successful
0807 22:45:23.181860 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260807 22:45:23.182413 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.178383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:57939 (local address 127.16.98.130:40377)
0807 22:45:23.178463 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:23.178466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:23.178547 (+ 81us) server_negotiation.cc:407] Connection header received
0807 22:45:23.178676 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:23.178678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:23.178691 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:23.178722 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:23.179019 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.179652 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.181138 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.181849 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.181935 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.181945 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:23.182278 (+ 333us) server_negotiation.cc:299] Negotiation successful
0807 22:45:23.182329 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 22:45:23.182413 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.178282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:40377 (local address 127.16.98.129:57939)
0807 22:45:23.178458 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:23.178478 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:23.178583 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:23.178791 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:23.178792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:23.178793 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:23.178896 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.178898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.179738 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.179739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.181032 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.181035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.182009 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.182010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.182189 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.182193 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:23.182260 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 22:45:23.182309 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260807 22:45:23.182575 18362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b404142abf4ce4b8e674e28cbe4010" candidate_uuid: "5d77444215074c87a27e8f76207a4d28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23fec5f96113472d8e22766fa6fa8d00" is_pre_election: true
I20260807 22:45:23.182686 18362 raft_consensus.cc:2454] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d77444215074c87a27e8f76207a4d28 in term 0.
I20260807 22:45:23.182752 18215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b404142abf4ce4b8e674e28cbe4010" candidate_uuid: "5d77444215074c87a27e8f76207a4d28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" is_pre_election: true
I20260807 22:45:23.182888 18215 raft_consensus.cc:2454] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d77444215074c87a27e8f76207a4d28 in term 0.
I20260807 22:45:23.182876 18001 leader_election.cc:302] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23fec5f96113472d8e22766fa6fa8d00, 5d77444215074c87a27e8f76207a4d28; no voters:
I20260807 22:45:23.183015 18440 raft_consensus.cc:2781] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:23.183059 18440 raft_consensus.cc:482] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:23.183080 18440 raft_consensus.cc:3037] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:23.183907 18440 raft_consensus.cc:504] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.184065 18440 leader_election.cc:288] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [CANDIDATE]: Term 1 election: Requested vote from peers c432cc2bc1fb4e55b11b9dc325b9e3f1 (127.16.98.130:40377), 23fec5f96113472d8e22766fa6fa8d00 (127.16.98.131:38101)
I20260807 22:45:23.184244 18362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b404142abf4ce4b8e674e28cbe4010" candidate_uuid: "5d77444215074c87a27e8f76207a4d28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23fec5f96113472d8e22766fa6fa8d00"
I20260807 22:45:23.184283 18215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b404142abf4ce4b8e674e28cbe4010" candidate_uuid: "5d77444215074c87a27e8f76207a4d28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1"
I20260807 22:45:23.184346 18362 raft_consensus.cc:3037] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:23.184381 18215 raft_consensus.cc:3037] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:23.185123 18362 raft_consensus.cc:2454] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d77444215074c87a27e8f76207a4d28 in term 1.
I20260807 22:45:23.185298 18001 leader_election.cc:302] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23fec5f96113472d8e22766fa6fa8d00, 5d77444215074c87a27e8f76207a4d28; no voters:
I20260807 22:45:23.185386 18440 raft_consensus.cc:2781] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:23.185540 18440 raft_consensus.cc:686] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [term 1 LEADER]: Becoming Leader. State: Replica: 5d77444215074c87a27e8f76207a4d28, State: Running, Role: LEADER
I20260807 22:45:23.185523 18215 raft_consensus.cc:2454] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d77444215074c87a27e8f76207a4d28 in term 1.
I20260807 22:45:23.185619 18440 consensus_queue.cc:237] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.186398 17908 catalog_manager.cc:5495] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d77444215074c87a27e8f76207a4d28 (127.16.98.129). New cstate: current_term: 1 leader_uuid: "5d77444215074c87a27e8f76207a4d28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } health_report { overall_health: UNKNOWN } } }
W20260807 22:45:23.213939 18263 tablet.cc:2369] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 07 22:45:23 dist-test-slave-rgb1 krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142722, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:23.237759 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.229431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36425 (local address 127.0.0.1:54046)
0807 22:45:23.229660 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:23.229669 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:23.229761 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:23.229926 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:23.229927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:23.230281 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:23.230328 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.230330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.231043 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.231045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.231560 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.231562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.231926 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.231928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.231980 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.232664 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:23.232669 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:23.234258 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:23.236337 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:23.236346 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:23.236349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:23.236695 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:23.236932 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:23.236933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:23.236933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:23.237016 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:23.237294 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:23.237304 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:23.237373 (+ 69us) client_negotiation.cc:769] Sending connection context
0807 22:45:23.237470 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 22:45:23.237618 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":126,"threads_started":1}
I20260807 22:45:23.237764 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.229536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.16.98.190:36425)
0807 22:45:23.229592 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:23.229594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:23.229745 (+ 151us) server_negotiation.cc:407] Connection header received
0807 22:45:23.229847 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:23.229851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:23.229861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:23.229923 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:23.230417 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.230960 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.231649 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.231839 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.231926 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.234358 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:23.234360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:23.234360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:23.234369 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:23.236190 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:23.236791 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:23.236792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:23.236793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:23.236859 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:23.237087 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:23.237088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:23.237088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:23.237220 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:23.237312 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:23.237496 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 22:45:23.237678 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":29}
I20260807 22:45:23.240666 17908 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:54046:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:45:23.241063 17908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:23.243075 18194 tablet_service.cc:1467] Processing CreateTablet for tablet ee5dfcafb4264958b5d91424c23a3a12 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16e4f0bf96234f299148f8c7ec7cb9a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:45:23.243093 18342 tablet_service.cc:1467] Processing CreateTablet for tablet ee5dfcafb4264958b5d91424c23a3a12 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16e4f0bf96234f299148f8c7ec7cb9a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:45:23.243103 18035 tablet_service.cc:1467] Processing CreateTablet for tablet ee5dfcafb4264958b5d91424c23a3a12 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16e4f0bf96234f299148f8c7ec7cb9a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:45:23.243279 18342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5dfcafb4264958b5d91424c23a3a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:23.243279 18035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5dfcafb4264958b5d91424c23a3a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:23.243283 18194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5dfcafb4264958b5d91424c23a3a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:23.246301 18438 tablet_bootstrap.cc:492] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1: Bootstrap starting.
I20260807 22:45:23.246382 18434 tablet_bootstrap.cc:492] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00: Bootstrap starting.
I20260807 22:45:23.246448 18435 tablet_bootstrap.cc:492] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28: Bootstrap starting.
I20260807 22:45:23.246750 18438 tablet_bootstrap.cc:654] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:23.246933 18434 tablet_bootstrap.cc:654] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:23.246933 18435 tablet_bootstrap.cc:654] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:23.247391 18438 tablet_bootstrap.cc:492] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1: No bootstrap required, opened a new log
I20260807 22:45:23.247465 18438 ts_tablet_manager.cc:1397] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:23.247625 18438 raft_consensus.cc:348] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.247687 18434 tablet_bootstrap.cc:492] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00: No bootstrap required, opened a new log
I20260807 22:45:23.247714 18438 raft_consensus.cc:374] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:23.247740 18438 raft_consensus.cc:729] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c432cc2bc1fb4e55b11b9dc325b9e3f1, State: Initialized, Role: FOLLOWER
I20260807 22:45:23.247731 18434 ts_tablet_manager.cc:1397] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:23.247792 18438 consensus_queue.cc:260] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.247895 18438 ts_tablet_manager.cc:1428] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260807 22:45:23.247920 18435 tablet_bootstrap.cc:492] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28: No bootstrap required, opened a new log
I20260807 22:45:23.247881 18434 raft_consensus.cc:348] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.247961 18434 raft_consensus.cc:374] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:23.247985 18434 raft_consensus.cc:729] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23fec5f96113472d8e22766fa6fa8d00, State: Initialized, Role: FOLLOWER
I20260807 22:45:23.247982 18435 ts_tablet_manager.cc:1397] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260807 22:45:23.247927 18442 tablet_replica.cc:439] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } }
I20260807 22:45:23.248026 18442 tablet_replica.cc:442] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:23.248061 18434 consensus_queue.cc:260] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.248159 18434 ts_tablet_manager.cc:1428] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:23.248149 18435 raft_consensus.cc:348] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.248222 18435 raft_consensus.cc:374] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:23.248245 18435 raft_consensus.cc:729] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d77444215074c87a27e8f76207a4d28, State: Initialized, Role: FOLLOWER
I20260807 22:45:23.248234 18441 tablet_replica.cc:439] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } }
I20260807 22:45:23.248294 18435 consensus_queue.cc:260] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.248343 18441 tablet_replica.cc:442] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:23.248407 18435 ts_tablet_manager.cc:1428] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260807 22:45:23.248523 18440 tablet_replica.cc:439] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } }
I20260807 22:45:23.248651 18440 tablet_replica.cc:442] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:23.375597 18441 raft_consensus.cc:482] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:23.375718 18441 raft_consensus.cc:504] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.376049 18441 leader_election.cc:288] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d77444215074c87a27e8f76207a4d28 (127.16.98.129:39687), c432cc2bc1fb4e55b11b9dc325b9e3f1 (127.16.98.130:40377)
I20260807 22:45:23.380219 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.376323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:39687 (local address 127.16.98.131:47615)
0807 22:45:23.376509 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:23.376525 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:23.376643 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:23.376870 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:23.376870 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:23.376872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:23.376956 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.376958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.377950 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.377951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.379373 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.379377 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.379967 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.379969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.380065 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.380069 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:23.380116 (+ 47us) client_negotiation.cc:240] Negotiation successful
0807 22:45:23.380150 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":3}
I20260807 22:45:23.380259 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.376501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:47615 (local address 127.16.98.129:39687)
0807 22:45:23.376547 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:23.376549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:23.376632 (+ 83us) server_negotiation.cc:407] Connection header received
0807 22:45:23.376782 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:23.376783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:23.376793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:23.376849 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:23.377079 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.377827 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.379516 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.379874 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.379936 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.379970 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:23.380142 (+ 172us) server_negotiation.cc:299] Negotiation successful
0807 22:45:23.380186 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 22:45:23.380622 18055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5dfcafb4264958b5d91424c23a3a12" candidate_uuid: "23fec5f96113472d8e22766fa6fa8d00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d77444215074c87a27e8f76207a4d28" is_pre_election: true
I20260807 22:45:23.380755 18055 raft_consensus.cc:2454] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23fec5f96113472d8e22766fa6fa8d00 in term 0.
I20260807 22:45:23.380955 18290 leader_election.cc:302] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23fec5f96113472d8e22766fa6fa8d00, 5d77444215074c87a27e8f76207a4d28; no voters:
I20260807 22:45:23.381042 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.376416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:59337 (local address 127.16.98.130:40377)
0807 22:45:23.376482 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:23.376484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:23.376607 (+ 123us) server_negotiation.cc:407] Connection header received
0807 22:45:23.376731 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:23.376732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:23.376745 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:23.376821 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:23.377079 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.377855 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.380014 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.380615 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.380688 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.380695 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:23.380892 (+ 197us) server_negotiation.cc:299] Negotiation successful
0807 22:45:23.380937 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260807 22:45:23.381124 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.376327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:40377 (local address 127.16.98.131:59337)
0807 22:45:23.376521 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:23.376532 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:23.376638 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:23.376855 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:23.376856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:23.376859 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:23.376973 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.376975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.377950 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.377951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.379883 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.379886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.380709 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.380711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.380808 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.380813 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:45:23.380883 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 22:45:23.380925 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":48,"thread_start_us":71,"threads_started":1}
I20260807 22:45:23.381162 18441 raft_consensus.cc:2781] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:23.381204 18441 raft_consensus.cc:482] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:23.381232 18441 raft_consensus.cc:3037] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:23.381390 18215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5dfcafb4264958b5d91424c23a3a12" candidate_uuid: "23fec5f96113472d8e22766fa6fa8d00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" is_pre_election: true
I20260807 22:45:23.381491 18215 raft_consensus.cc:2454] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23fec5f96113472d8e22766fa6fa8d00 in term 0.
I20260807 22:45:23.382082 18441 raft_consensus.cc:504] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.382242 18441 leader_election.cc:288] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [CANDIDATE]: Term 1 election: Requested vote from peers 5d77444215074c87a27e8f76207a4d28 (127.16.98.129:39687), c432cc2bc1fb4e55b11b9dc325b9e3f1 (127.16.98.130:40377)
I20260807 22:45:23.382388 18215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5dfcafb4264958b5d91424c23a3a12" candidate_uuid: "23fec5f96113472d8e22766fa6fa8d00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1"
I20260807 22:45:23.382409 18055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5dfcafb4264958b5d91424c23a3a12" candidate_uuid: "23fec5f96113472d8e22766fa6fa8d00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d77444215074c87a27e8f76207a4d28"
I20260807 22:45:23.382475 18215 raft_consensus.cc:3037] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:23.382501 18055 raft_consensus.cc:3037] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:23.383165 18055 raft_consensus.cc:2454] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23fec5f96113472d8e22766fa6fa8d00 in term 1.
I20260807 22:45:23.383347 18215 raft_consensus.cc:2454] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23fec5f96113472d8e22766fa6fa8d00 in term 1.
I20260807 22:45:23.383386 18290 leader_election.cc:302] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23fec5f96113472d8e22766fa6fa8d00, 5d77444215074c87a27e8f76207a4d28; no voters:
I20260807 22:45:23.383486 18441 raft_consensus.cc:2781] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:23.383648 18441 raft_consensus.cc:686] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [term 1 LEADER]: Becoming Leader. State: Replica: 23fec5f96113472d8e22766fa6fa8d00, State: Running, Role: LEADER
I20260807 22:45:23.383752 18441 consensus_queue.cc:237] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } }
I20260807 22:45:23.384181 18477 tablet_replica.cc:439] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00: TxnStatusTablet state changed. Reason: New leader 23fec5f96113472d8e22766fa6fa8d00. Latest consensus state: current_term: 1 leader_uuid: "23fec5f96113472d8e22766fa6fa8d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } }
I20260807 22:45:23.384274 18477 tablet_replica.cc:442] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:45:23.384459 18479 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:45:23.384590 18479 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:45:23.384792 17907 catalog_manager.cc:5495] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23fec5f96113472d8e22766fa6fa8d00 (127.16.98.131). New cstate: current_term: 1 leader_uuid: "23fec5f96113472d8e22766fa6fa8d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } health_report { overall_health: HEALTHY } } }
I20260807 22:45:23.399287 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.394380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.16.98.131:38101)
0807 22:45:23.394432 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:23.394435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:23.394445 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:45:23.394527 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:23.394528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:23.394548 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:23.394647 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:23.394828 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.395947 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.397982 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.398330 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.398435 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.398759 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:23.398943 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:23.399031 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:23.399158 (+ 127us) server_negotiation.cc:299] Negotiation successful
0807 22:45:23.399186 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 22:45:23.399406 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.394281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:38101 (local address 127.0.0.1:55262)
0807 22:45:23.394377 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:23.394390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:23.394450 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:23.394659 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:23.394660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:23.394662 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:45:23.394752 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.394756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.396102 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.396104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.397853 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.397858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.398452 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.398454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.398560 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.398617 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:45:23.399051 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:45:23.399061 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 22:45:23.399154 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 22:45:23.399250 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260807 22:45:23.410048 18215 raft_consensus.cc:1261] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 1 FOLLOWER]: Refusing update from remote peer 23fec5f96113472d8e22766fa6fa8d00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:23.410364 18055 raft_consensus.cc:1261] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28 [term 1 FOLLOWER]: Refusing update from remote peer 23fec5f96113472d8e22766fa6fa8d00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:23.410393 18441 consensus_queue.cc:1035] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:23.410648 18441 consensus_queue.cc:1035] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:23.411711 18442 tablet_replica.cc:439] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1: TxnStatusTablet state changed. Reason: New leader 23fec5f96113472d8e22766fa6fa8d00. Latest consensus state: current_term: 1 leader_uuid: "23fec5f96113472d8e22766fa6fa8d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } }
I20260807 22:45:23.411811 18442 tablet_replica.cc:442] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:23.411756 18440 tablet_replica.cc:439] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28: TxnStatusTablet state changed. Reason: New leader 23fec5f96113472d8e22766fa6fa8d00. Latest consensus state: current_term: 1 leader_uuid: "23fec5f96113472d8e22766fa6fa8d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } }
I20260807 22:45:23.411859 18440 tablet_replica.cc:442] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:23.413280 18477 tablet_replica.cc:439] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23fec5f96113472d8e22766fa6fa8d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } }
I20260807 22:45:23.413374 18477 tablet_replica.cc:442] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:45:23.413420 18477 tablet_replica.cc:439] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23fec5f96113472d8e22766fa6fa8d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } }
I20260807 22:45:23.413457 18477 tablet_replica.cc:442] T ee5dfcafb4264958b5d91424c23a3a12 P 23fec5f96113472d8e22766fa6fa8d00: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:45:23.413697 18442 tablet_replica.cc:439] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23fec5f96113472d8e22766fa6fa8d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } }
I20260807 22:45:23.413796 18442 tablet_replica.cc:442] T ee5dfcafb4264958b5d91424c23a3a12 P c432cc2bc1fb4e55b11b9dc325b9e3f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:23.414284 18449 mvcc.cc:205] Tried to move back new op lower bound from 7316040595083956224 to 7316040594980548608. Current Snapshot: MvccSnapshot[applied={T|T < 7316040595083956224}]
I20260807 22:45:23.414413 18440 tablet_replica.cc:439] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23fec5f96113472d8e22766fa6fa8d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d77444215074c87a27e8f76207a4d28" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39687 } } peers { permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 } } peers { permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 } } }
I20260807 22:45:23.414491 18440 tablet_replica.cc:442] T ee5dfcafb4264958b5d91424c23a3a12 P 5d77444215074c87a27e8f76207a4d28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:23.420033 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.416416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.16.98.129:39687)
0807 22:45:23.416465 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:23.416467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:23.416500 (+ 33us) server_negotiation.cc:407] Connection header received
0807 22:45:23.416542 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:23.416543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:23.416597 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:23.416672 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:23.416818 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.417670 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.418897 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.419239 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.419326 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.419492 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:23.419671 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:23.419755 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:23.419872 (+ 117us) server_negotiation.cc:299] Negotiation successful
0807 22:45:23.419915 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 22:45:23.426612 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.423329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:55125 (local address 127.16.98.131:38101)
0807 22:45:23.423387 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:23.423389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:23.423495 (+ 106us) server_negotiation.cc:407] Connection header received
0807 22:45:23.423529 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:23.423530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:23.423540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:23.423598 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:23.423793 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.424377 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.425805 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.426021 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.426093 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.426338 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:23.426370 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:23.426418 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:23.426516 (+ 98us) server_negotiation.cc:299] Negotiation successful
0807 22:45:23.426548 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260807 22:45:23.426681 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.423220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:38101 (local address 127.16.98.129:55125)
0807 22:45:23.423391 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:23.423404 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:23.423479 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:23.423603 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:23.423603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:23.423605 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:45:23.423684 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.423685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.424478 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.424479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.425622 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.425625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.426113 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.426114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.426194 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.426220 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:45:23.426449 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:45:23.426451 (+ 2us) client_negotiation.cc:769] Sending connection context
0807 22:45:23.426511 (+ 60us) client_negotiation.cc:240] Negotiation successful
0807 22:45:23.426564 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260807 22:45:23.429890 18215 raft_consensus.cc:1261] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 1 FOLLOWER]: Refusing update from remote peer 5d77444215074c87a27e8f76207a4d28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:23.429986 18362 raft_consensus.cc:1261] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00 [term 1 FOLLOWER]: Refusing update from remote peer 5d77444215074c87a27e8f76207a4d28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:23.430270 18440 consensus_queue.cc:1035] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c432cc2bc1fb4e55b11b9dc325b9e3f1" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:23.430436 18465 consensus_queue.cc:1035] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23fec5f96113472d8e22766fa6fa8d00" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:23.432482 18448 mvcc.cc:205] Tried to move back new op lower bound from 7316040595166756864 to 7316040594168881152. Current Snapshot: MvccSnapshot[applied={T|T < 7316040595166756864}]
I20260807 22:45:23.432747 18447 mvcc.cc:205] Tried to move back new op lower bound from 7316040595166756864 to 7316040594168881152. Current Snapshot: MvccSnapshot[applied={T|T < 7316040595166756864}]
I20260807 22:45:23.444572 18491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.437044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:39687 (local address 127.16.98.131:35357)
0807 22:45:23.440705 (+ 3661us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:23.440720 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:23.440795 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:23.440952 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:23.440953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:23.440974 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:45:23.441083 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.441085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.442163 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.442164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.443268 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:23.443271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.443686 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.443687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:23.443766 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.443790 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:45:23.444273 (+ 483us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:45:23.444276 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:45:23.444373 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 22:45:23.444428 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3606,"thread_start_us":102,"threads_started":1}
I20260807 22:45:23.444614 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.437111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:35357 (local address 127.16.98.129:39687)
0807 22:45:23.437155 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:23.437157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:23.440801 (+ 3644us) server_negotiation.cc:407] Connection header received
0807 22:45:23.440867 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:23.440868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:23.440880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:23.440927 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:23.441197 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.442052 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.443372 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.443599 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.443664 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.444057 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:23.444104 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:23.444173 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:23.444467 (+ 294us) server_negotiation.cc:299] Negotiation successful
0807 22:45:23.444498 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260807 22:45:23.454726 17907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:23.454859 17907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:23.455637 17907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d585397cd91464db4d7fb87fdaa6ecf: Sending DeleteTablet for 3 replicas of tablet e5b404142abf4ce4b8e674e28cbe4010
I20260807 22:45:23.455900 18035 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b404142abf4ce4b8e674e28cbe4010 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:23 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:60558
I20260807 22:45:23.455973 18194 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b404142abf4ce4b8e674e28cbe4010 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:23 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:57470
I20260807 22:45:23.456020 18458 tablet_replica.cc:330] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28: stopping tablet replica
I20260807 22:45:23.456075 18458 raft_consensus.cc:2229] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:45:23.456091 18459 tablet_replica.cc:330] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1: stopping tablet replica
I20260807 22:45:23.456156 18459 raft_consensus.cc:2229] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:23.456180 18458 raft_consensus.cc:2258] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:23.456208 18459 raft_consensus.cc:2258] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:23.456216 18342 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b404142abf4ce4b8e674e28cbe4010 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:23 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:55260
I20260807 22:45:23.456341 18457 tablet_replica.cc:330] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00: stopping tablet replica
I20260807 22:45:23.456426 18457 raft_consensus.cc:2229] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:23.456477 18457 raft_consensus.cc:2258] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:23.456694 18459 ts_tablet_manager.cc:1905] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:23.458451 18458 ts_tablet_manager.cc:1905] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:23.458506 18459 ts_tablet_manager.cc:1918] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:45:23.458571 18459 log.cc:1196] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/e5b404142abf4ce4b8e674e28cbe4010
I20260807 22:45:23.458830 18459 ts_tablet_manager.cc:1939] T e5b404142abf4ce4b8e674e28cbe4010 P c432cc2bc1fb4e55b11b9dc325b9e3f1: Deleting consensus metadata
I20260807 22:45:23.459188 17894 catalog_manager.cc:4841] TS c432cc2bc1fb4e55b11b9dc325b9e3f1 (127.16.98.130:40377): tablet e5b404142abf4ce4b8e674e28cbe4010 (table test-table [id=8a960f7b57354b3ea0b3cbc7127bdedf]) successfully deleted
I20260807 22:45:23.460165 18458 ts_tablet_manager.cc:1918] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:45:23.460222 18458 log.cc:1196] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-0/wal/wals/e5b404142abf4ce4b8e674e28cbe4010
I20260807 22:45:23.460467 18458 ts_tablet_manager.cc:1939] T e5b404142abf4ce4b8e674e28cbe4010 P 5d77444215074c87a27e8f76207a4d28: Deleting consensus metadata
I20260807 22:45:23.460841 17893 catalog_manager.cc:4841] TS 5d77444215074c87a27e8f76207a4d28 (127.16.98.129:39687): tablet e5b404142abf4ce4b8e674e28cbe4010 (table test-table [id=8a960f7b57354b3ea0b3cbc7127bdedf]) successfully deleted
I20260807 22:45:23.460991 18457 ts_tablet_manager.cc:1905] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:23.462764 18457 ts_tablet_manager.cc:1918] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:45:23.462827 18457 log.cc:1196] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/e5b404142abf4ce4b8e674e28cbe4010
I20260807 22:45:23.463070 18457 ts_tablet_manager.cc:1939] T e5b404142abf4ce4b8e674e28cbe4010 P 23fec5f96113472d8e22766fa6fa8d00: Deleting consensus metadata
I20260807 22:45:23.463650 17894 catalog_manager.cc:4841] TS 23fec5f96113472d8e22766fa6fa8d00 (127.16.98.131:38101): tablet e5b404142abf4ce4b8e674e28cbe4010 (table test-table [id=8a960f7b57354b3ea0b3cbc7127bdedf]) successfully deleted
Aug 07 22:45:23 dist-test-slave-rgb1 krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142722, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.98.129@KRBTEST.COM
I20260807 22:45:23.482646 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.468952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.16.98.129:39687)
0807 22:45:23.469021 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:23.469024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:23.469276 (+ 252us) server_negotiation.cc:407] Connection header received
0807 22:45:23.469386 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:23.469387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:23.469400 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:23.469493 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:23.470508 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.471345 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.472473 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.472892 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.472994 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.478246 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:23.478258 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:23.478259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:23.478269 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:23.481051 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:23.481538 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:23.481542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:23.481543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:23.481636 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:23.481892 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:23.481893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:23.481894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:23.482105 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:23.482296 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:23.482406 (+ 110us) server_negotiation.cc:299] Negotiation successful
0807 22:45:23.482504 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260807 22:45:23.483165 18112 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60576
I20260807 22:45:23.493723 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:23.484434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.16.98.190:36425)
0807 22:45:23.484493 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:23.484495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:23.484531 (+ 36us) server_negotiation.cc:407] Connection header received
0807 22:45:23.484712 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:23.484713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:23.484725 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:23.484797 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:23.485540 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.486434 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.487559 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:23.487904 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:23.487997 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:23.489859 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:23.489861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:23.489862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:23.489871 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:23.491841 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:23.492318 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:23.492319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:23.492319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:23.492389 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:23.492695 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:23.492696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:23.492696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:23.492859 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:23.492966 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:23.493303 (+ 337us) server_negotiation.cc:299] Negotiation successful
0807 22:45:23.493421 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260807 22:45:23.494056 17907 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54060
I20260807 22:45:23.495240 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 17983
I20260807 22:45:23.506927 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 18128
I20260807 22:45:23.516253 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 18272
I20260807 22:45:23.524886 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 17876
2026-08-07T22:45:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2640 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:45:23 dist-test-slave-rgb1 krb5kdc[18501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:45:23 dist-test-slave-rgb1 krb5kdc[18501](info): set up 2 sockets
Aug 07 22:45:23 dist-test-slave-rgb1 krb5kdc[18501](info): commencing operation
krb5kdc: starting...
W20260807 22:45:24.681011 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:45:24 dist-test-slave-rgb1 krb5kdc[18501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:45:24.825743 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:42895
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:42895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:24.917284 18515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:24.917552 18515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:24.917595 18515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:24.918784 18515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:24.918959 18515 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 22:45:24.919015 18515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:24.919044 18515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:24.919073 18515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:24.919099 18515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:24.920754 18515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:42895
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:42895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:24.921072 18515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:24.921332 18515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:24.926795 18523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:24.926788 18520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:24.926807 18521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:24.927356 18515 server_base.cc:1034] running on GCE node
I20260807 22:45:24.927644 18515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:24.927963 18515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:24.929123 18515 hybrid_clock.cc:648] HybridClock initialized: now 1786142724929044 us; error 96 us; skew 500 ppm
Aug 07 22:45:24 dist-test-slave-rgb1 krb5kdc[18501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142724, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:24.932307 18515 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:45:24.933010 18515 webserver.cc:460] Webserver started at http://127.16.98.190:41245/ using document root <none> and password file <none>
I20260807 22:45:24.933328 18515 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:24.933409 18515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:24.933590 18515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:24.934787 18515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "37c4d63f6a6a42dda3125546d1cba386"
format_stamp: "Formatted at 2026-08-07 22:45:24 on dist-test-slave-rgb1"
server_key: "aeb03b4acbdd47bf24c5c01aea396a96"
server_key_iv: "8083509b424dda47c6745452840597a2"
server_key_version: "encryptionkey@0"
I20260807 22:45:24.935158 18515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "37c4d63f6a6a42dda3125546d1cba386"
format_stamp: "Formatted at 2026-08-07 22:45:24 on dist-test-slave-rgb1"
server_key: "aeb03b4acbdd47bf24c5c01aea396a96"
server_key_iv: "8083509b424dda47c6745452840597a2"
server_key_version: "encryptionkey@0"
I20260807 22:45:24.937788 18515 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.003s
I20260807 22:45:24.938886 18531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:24.939451 18515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:24.939620 18515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "37c4d63f6a6a42dda3125546d1cba386"
format_stamp: "Formatted at 2026-08-07 22:45:24 on dist-test-slave-rgb1"
server_key: "aeb03b4acbdd47bf24c5c01aea396a96"
server_key_iv: "8083509b424dda47c6745452840597a2"
server_key_version: "encryptionkey@0"
I20260807 22:45:24.939822 18515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:24.954805 18515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:24.957916 18515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:24.958076 18515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:24.965242 18515 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:42895
I20260807 22:45:24.965411 18601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:42895 every 8 connection(s)
I20260807 22:45:24.965762 18515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:24.966589 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:24.969192 18602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386: Bootstrap starting.
I20260807 22:45:24.969856 18602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:24.970180 18602 log.cc:824] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:24.971346 18602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386: No bootstrap required, opened a new log
I20260807 22:45:24.973196 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 18515
I20260807 22:45:24.973276 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:24.973506 16778 external_mini_cluster.cc:1428] Setting key 849a1160e1f76d950eefea30c01340bc
I20260807 22:45:24.974565 18602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c4d63f6a6a42dda3125546d1cba386" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42895 } }
I20260807 22:45:24.974771 18602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:24.974828 18602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37c4d63f6a6a42dda3125546d1cba386, State: Initialized, Role: FOLLOWER
I20260807 22:45:24.975004 18602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c4d63f6a6a42dda3125546d1cba386" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42895 } }
I20260807 22:45:24.975111 18602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:24.975150 18602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:24.975214 18602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:24.976078 18602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c4d63f6a6a42dda3125546d1cba386" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42895 } }
I20260807 22:45:24.976186 18602 leader_election.cc:302] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37c4d63f6a6a42dda3125546d1cba386; no voters:
I20260807 22:45:24.976415 18602 leader_election.cc:288] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:24.976469 18607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:24.976665 18607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [term 1 LEADER]: Becoming Leader. State: Replica: 37c4d63f6a6a42dda3125546d1cba386, State: Running, Role: LEADER
I20260807 22:45:24.976759 18602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:24.976770 18607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c4d63f6a6a42dda3125546d1cba386" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42895 } }
I20260807 22:45:24.977176 18608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37c4d63f6a6a42dda3125546d1cba386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c4d63f6a6a42dda3125546d1cba386" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42895 } } }
I20260807 22:45:24.977293 18608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:24.977583 18609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37c4d63f6a6a42dda3125546d1cba386. Latest consensus state: current_term: 1 leader_uuid: "37c4d63f6a6a42dda3125546d1cba386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c4d63f6a6a42dda3125546d1cba386" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42895 } } }
I20260807 22:45:24.977890 18609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:24.978127 18613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:24.978760 18613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:24.980782 18613 catalog_manager.cc:1347] Generated new cluster ID: 09d0fc42e8f54ef2aed02c89c9b95811
I20260807 22:45:24.980842 18613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 22:45:24 dist-test-slave-rgb1 krb5kdc[18501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:24.987721 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:24.974219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.16.98.190:42895)
0807 22:45:24.974419 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:24.974427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:24.974455 (+ 28us) server_negotiation.cc:407] Connection header received
0807 22:45:24.974591 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:24.974597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:24.974653 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:24.974832 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:24.975810 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:24.977440 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:24.978759 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:24.979463 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:24.979598 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:24.983443 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:24.983449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:24.983449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:24.983463 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:24.985804 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:24.986272 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:24.986275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:24.986276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:24.986406 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:24.986637 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:24.986638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:24.986638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:24.986986 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:24.987158 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:24.987293 (+ 135us) server_negotiation.cc:299] Negotiation successful
0807 22:45:24.987450 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":112,"threads_started":1}
I20260807 22:45:25.007572 18613 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:25.008231 18613 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:25.018276 18613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386: Generated new TSK 0
I20260807 22:45:25.018535 18613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:45:25.057642 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:42895
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:25.144964 18629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:25.145226 18629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:25.145258 18629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:25.146474 18629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:25.146522 18629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:25.146544 18629 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:45:25.146556 18629 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:45:25.146575 18629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:45:25.148120 18629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.98.190:42895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:25.148381 18629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:25.148712 18629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:25.154484 18634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:25.154671 18635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:25.154562 18637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:25.155331 18629 server_base.cc:1034] running on GCE node
I20260807 22:45:25.155586 18629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:25.155916 18629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:25.157161 18629 hybrid_clock.cc:648] HybridClock initialized: now 1786142725157137 us; error 86 us; skew 500 ppm
Aug 07 22:45:25 dist-test-slave-rgb1 krb5kdc[18501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142725, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:25.161269 18629 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:45:25.161814 18629 webserver.cc:460] Webserver started at http://127.16.98.129:38927/ using document root <none> and password file <none>
I20260807 22:45:25.162106 18629 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:25.162204 18629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:25.162371 18629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:25.163502 18629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "6b1d94c7563d45eebd68ffa45fa74baf"
format_stamp: "Formatted at 2026-08-07 22:45:25 on dist-test-slave-rgb1"
server_key: "a45b40999300a9efa00ce7cfda5dfc59"
server_key_iv: "e6939a13fb18aea537ce1ba73cdb153d"
server_key_version: "encryptionkey@0"
I20260807 22:45:25.163895 18629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "6b1d94c7563d45eebd68ffa45fa74baf"
format_stamp: "Formatted at 2026-08-07 22:45:25 on dist-test-slave-rgb1"
server_key: "a45b40999300a9efa00ce7cfda5dfc59"
server_key_iv: "e6939a13fb18aea537ce1ba73cdb153d"
server_key_version: "encryptionkey@0"
I20260807 22:45:25.165545 18629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:45:25.166543 18645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:25.166905 18629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:25.167040 18629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "6b1d94c7563d45eebd68ffa45fa74baf"
format_stamp: "Formatted at 2026-08-07 22:45:25 on dist-test-slave-rgb1"
server_key: "a45b40999300a9efa00ce7cfda5dfc59"
server_key_iv: "e6939a13fb18aea537ce1ba73cdb153d"
server_key_version: "encryptionkey@0"
I20260807 22:45:25.167135 18629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:25.193089 18629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:25.196293 18629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:25.196514 18629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:25.197134 18629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:25.197201 18629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:25.197247 18629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:25.197291 18629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 22:45:25 dist-test-slave-rgb1 krb5kdc[18501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142725, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:25.206422 18629 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:42677
I20260807 22:45:25.206584 18765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:42677 every 8 connection(s)
I20260807 22:45:25.206881 18629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:45:25.209496 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.197806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:43545 (local address 127.16.98.190:42895)
0807 22:45:25.197909 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.197911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.198151 (+ 240us) server_negotiation.cc:407] Connection header received
0807 22:45:25.198747 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.198748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.198769 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.198860 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:25.199931 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.200913 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.202182 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.202436 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.202527 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.205653 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:25.205655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:25.205655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:25.205665 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:25.207608 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.208080 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.208082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.208083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.208204 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.208651 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.208652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.208652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.208815 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:25.208928 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.209307 (+ 379us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.209413 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 22:45:25.209641 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.197755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:42895 (local address 127.16.98.129:43545)
0807 22:45:25.197952 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.198004 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.198599 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.199108 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.199113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.199495 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:25.199829 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.199836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.201057 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.201058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.202044 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.202046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.202599 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.202600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.202731 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.203435 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:25.203447 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:25.205513 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:25.207694 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:25.207699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:25.207700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:25.207928 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:25.208322 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:25.208323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:25.208323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:25.208467 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:25.208925 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:25.208932 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:25.209103 (+ 171us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.209219 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.209396 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
I20260807 22:45:25.210314 18766 heartbeater.cc:347] Connected to a master server at 127.16.98.190:42895
I20260807 22:45:25.210500 18766 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:25.210752 18766 heartbeater.cc:511] Master 127.16.98.190:42895 requested a full tablet report, sending...
I20260807 22:45:25.211503 18553 ts_manager.cc:194] Registered new tserver with Master: 6b1d94c7563d45eebd68ffa45fa74baf (127.16.98.129:42677)
I20260807 22:45:25.212184 18553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:43545
I20260807 22:45:25.214290 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 18629
I20260807 22:45:25.214430 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:45:25.214658 16778 external_mini_cluster.cc:1428] Setting key 8e716ab3b92a83c58a26cde5f077d673
I20260807 22:45:25.218254 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.211533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:33969 (local address 127.16.98.190:42895)
0807 22:45:25.211587 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.211589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.211642 (+ 53us) server_negotiation.cc:407] Connection header received
0807 22:45:25.211717 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.211717 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.211727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.211771 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:25.212118 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.212767 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.213499 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.213714 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.213791 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.215231 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:25.215232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:25.215233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:25.215239 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:25.216998 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.217397 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.217398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.217399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.217463 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.217668 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.217669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.217670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.217785 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:25.217856 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.218071 (+ 215us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.218156 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260807 22:45:25.218256 18773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.211457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:42895 (local address 127.16.98.129:33969)
0807 22:45:25.211589 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.211600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.211659 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.211785 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.211785 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.211983 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:25.212035 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.212037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.212866 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.212868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.213407 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.213409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.213790 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.213791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.213872 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.214613 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:25.214619 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:25.215129 (+ 510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:25.217127 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:25.217128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:25.217129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:25.217316 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:25.217530 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:25.217531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:25.217532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:25.217594 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:25.217915 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:25.217916 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:25.217967 (+ 51us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.218044 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.218158 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:45:25.245146 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:42895
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:25.333209 18776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:25.333444 18776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:25.333472 18776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:25.334677 18776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:25.334731 18776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:25.334754 18776 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:45:25.334769 18776 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:45:25.334789 18776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:45:25.336360 18776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.98.190:42895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:25.336653 18776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:25.336946 18776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:25.342481 18784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:25.342607 18781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:25.342715 18776 server_base.cc:1034] running on GCE node
W20260807 22:45:25.342484 18782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:25.343086 18776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:25.343384 18776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:25.344619 18776 hybrid_clock.cc:648] HybridClock initialized: now 1786142725344586 us; error 41 us; skew 500 ppm
Aug 07 22:45:25 dist-test-slave-rgb1 krb5kdc[18501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142725, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:25.347551 18776 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:45:25.348155 18776 webserver.cc:460] Webserver started at http://127.16.98.130:39483/ using document root <none> and password file <none>
I20260807 22:45:25.348424 18776 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:25.348505 18776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:25.348713 18776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:25.349849 18776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "a7483aaa4b0c41ac9e67ebbf320f5663"
format_stamp: "Formatted at 2026-08-07 22:45:25 on dist-test-slave-rgb1"
server_key: "cdcb64a1620e360a117db4c8374c676d"
server_key_iv: "f94873e528cbbe7c491a83f896a09d23"
server_key_version: "encryptionkey@0"
I20260807 22:45:25.350281 18776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "a7483aaa4b0c41ac9e67ebbf320f5663"
format_stamp: "Formatted at 2026-08-07 22:45:25 on dist-test-slave-rgb1"
server_key: "cdcb64a1620e360a117db4c8374c676d"
server_key_iv: "f94873e528cbbe7c491a83f896a09d23"
server_key_version: "encryptionkey@0"
I20260807 22:45:25.351775 18776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:45:25.352813 18792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:25.353060 18776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:25.353181 18776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "a7483aaa4b0c41ac9e67ebbf320f5663"
format_stamp: "Formatted at 2026-08-07 22:45:25 on dist-test-slave-rgb1"
server_key: "cdcb64a1620e360a117db4c8374c676d"
server_key_iv: "f94873e528cbbe7c491a83f896a09d23"
server_key_version: "encryptionkey@0"
I20260807 22:45:25.353264 18776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:25.377820 18776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:25.380316 18776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:25.380497 18776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:25.381191 18776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:25.381245 18776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:25.381279 18776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:25.381304 18776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 22:45:25 dist-test-slave-rgb1 krb5kdc[18501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142725, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:25.389045 18776 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:35979
I20260807 22:45:25.389493 18912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:35979 every 8 connection(s)
I20260807 22:45:25.389515 18776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:45:25.391907 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 18776
I20260807 22:45:25.392040 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:45:25.392140 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.381622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:58423 (local address 127.16.98.190:42895)
0807 22:45:25.381677 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.381679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.381981 (+ 302us) server_negotiation.cc:407] Connection header received
0807 22:45:25.382500 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.382501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.382515 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.382606 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:25.383260 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.384128 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.385345 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.385711 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.385807 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.388770 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:25.388772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:25.388772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:25.388781 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:25.390476 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.390934 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.390936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.390936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.391022 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.391295 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.391297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.391297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.391427 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:25.391506 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.391860 (+ 354us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.391942 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 22:45:25.392254 16778 external_mini_cluster.cc:1428] Setting key e7e14e8b48241c203b579ee21d664d47
I20260807 22:45:25.392252 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.381634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:42895 (local address 127.16.98.130:58423)
0807 22:45:25.381835 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.381878 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.382397 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.382691 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.382695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.382974 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:25.383181 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.383186 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.384231 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.384232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.385274 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.385276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.385850 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.385851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.386006 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.386608 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:25.386624 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:25.388664 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:25.390586 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:25.390595 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:25.390597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:25.390848 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:25.391099 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:25.391100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:25.391100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:25.391216 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:25.391499 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:25.391507 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:25.391654 (+ 147us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.391803 (+ 149us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.391999 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":118,"threads_started":1}
I20260807 22:45:25.392815 18913 heartbeater.cc:347] Connected to a master server at 127.16.98.190:42895
I20260807 22:45:25.392903 18913 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:25.393131 18913 heartbeater.cc:511] Master 127.16.98.190:42895 requested a full tablet report, sending...
I20260807 22:45:25.395022 18554 ts_manager.cc:194] Registered new tserver with Master: a7483aaa4b0c41ac9e67ebbf320f5663 (127.16.98.130:35979)
I20260807 22:45:25.395566 18554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:58423
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
I20260807 22:45:25.404961 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.394060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:42895 (local address 127.16.98.130:33307)
0807 22:45:25.394254 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.394265 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.394326 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.396765 (+ 2439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.396767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.397088 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:25.397175 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.397177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.398437 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.398438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.399517 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.399520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.400640 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.400642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.400760 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.401390 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:25.401396 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:25.401812 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:25.403718 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:25.403719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:25.403719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:25.403901 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:25.404128 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:25.404129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:25.404129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:25.404192 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:25.404605 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:25.404606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:25.404654 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.404742 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.404855 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260807 22:45:25.404958 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.394895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:33307 (local address 127.16.98.190:42895)
0807 22:45:25.394959 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.394961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.394970 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:45:25.394992 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.395000 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.395009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.395072 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:25.397278 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.398307 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.399618 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.399993 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.400076 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.401910 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:25.401912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:25.401912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:25.401920 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:25.403625 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.403992 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.403993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.403994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.404053 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.404267 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.404268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.404268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.404478 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:25.404588 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.404770 (+ 182us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.404852 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:45:25.428864 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:42895
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:25.517825 18923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:25.518124 18923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:25.518173 18923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:25.519383 18923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:25.519426 18923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:25.519447 18923 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:45:25.519459 18923 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:45:25.519500 18923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:45:25.521083 18923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.98.190:42895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:25.521339 18923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:25.521595 18923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:25.526973 18932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:25.527006 18929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:25.527558 18928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:25.527596 18923 server_base.cc:1034] running on GCE node
I20260807 22:45:25.527817 18923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:25.528081 18923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:25.529255 18923 hybrid_clock.cc:648] HybridClock initialized: now 1786142725529242 us; error 35 us; skew 500 ppm
Aug 07 22:45:25 dist-test-slave-rgb1 krb5kdc[18501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142725, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:25.531945 18923 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:45:25.532455 18923 webserver.cc:460] Webserver started at http://127.16.98.131:33869/ using document root <none> and password file <none>
I20260807 22:45:25.532740 18923 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:25.532811 18923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:25.532974 18923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:25.534106 18923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "48670158931f461baaaf58a56e241499"
format_stamp: "Formatted at 2026-08-07 22:45:25 on dist-test-slave-rgb1"
server_key: "fdbb46a2c2101fedca4687de7c3c62f2"
server_key_iv: "8b7243acedb5a4b11aa103517774309c"
server_key_version: "encryptionkey@0"
I20260807 22:45:25.534473 18923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "48670158931f461baaaf58a56e241499"
format_stamp: "Formatted at 2026-08-07 22:45:25 on dist-test-slave-rgb1"
server_key: "fdbb46a2c2101fedca4687de7c3c62f2"
server_key_iv: "8b7243acedb5a4b11aa103517774309c"
server_key_version: "encryptionkey@0"
I20260807 22:45:25.536026 18923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:45:25.537010 18939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:25.537297 18923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:25.537408 18923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "48670158931f461baaaf58a56e241499"
format_stamp: "Formatted at 2026-08-07 22:45:25 on dist-test-slave-rgb1"
server_key: "fdbb46a2c2101fedca4687de7c3c62f2"
server_key_iv: "8b7243acedb5a4b11aa103517774309c"
server_key_version: "encryptionkey@0"
I20260807 22:45:25.537508 18923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:25.561889 18923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:25.564515 18923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:25.564790 18923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:25.565614 18923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:25.565662 18923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:25.565689 18923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:25.565704 18923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 22:45:25 dist-test-slave-rgb1 krb5kdc[18501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142725, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:25.573680 18923 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:44073
I20260807 22:45:25.573966 19059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:44073 every 8 connection(s)
I20260807 22:45:25.574148 18923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:45:25.575714 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 18923
I20260807 22:45:25.575832 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:45:25.576031 16778 external_mini_cluster.cc:1428] Setting key d7916c88e83a35c7e06cadf4561648d8
I20260807 22:45:25.578089 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.566084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:39419 (local address 127.16.98.190:42895)
0807 22:45:25.566160 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.566163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.566519 (+ 356us) server_negotiation.cc:407] Connection header received
0807 22:45:25.567110 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.567112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.567130 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.567227 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:25.568366 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.569123 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.570504 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.570819 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.570904 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.574190 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:25.574192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:25.574193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:25.574202 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:25.576279 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.576908 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.576910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.576911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.576986 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.577260 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.577260 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.577261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.577388 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:25.577450 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.577894 (+ 444us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.578007 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 22:45:25.578380 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.566159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:42895 (local address 127.16.98.131:39419)
0807 22:45:25.566345 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.566387 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.566943 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.567385 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.567390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.567898 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:25.568255 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.568263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.569241 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.569242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.570406 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.570408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.570949 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.570951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.571099 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.571799 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:25.571808 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:25.574089 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:25.576362 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:25.576370 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:25.576372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:25.576801 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:25.577071 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:25.577071 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:25.577072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:25.577189 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:25.577529 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:25.577534 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:25.577739 (+ 205us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.577838 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.578084 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":113,"threads_started":1}
I20260807 22:45:25.578893 19060 heartbeater.cc:347] Connected to a master server at 127.16.98.190:42895
I20260807 22:45:25.578994 19060 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:25.579200 19060 heartbeater.cc:511] Master 127.16.98.190:42895 requested a full tablet report, sending...
I20260807 22:45:25.579540 18554 ts_manager.cc:194] Registered new tserver with Master: 48670158931f461baaaf58a56e241499 (127.16.98.131:44073)
I20260807 22:45:25.580021 18554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:39419
I20260807 22:45:25.586212 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.580146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:42895 (local address 127.16.98.131:51033)
0807 22:45:25.580316 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.580328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.580398 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.580549 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.580550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.580792 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:25.580897 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.580899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.581797 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.581798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.582372 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.582375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.582722 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.582723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.582785 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.583309 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:25.583314 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:25.583631 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:25.585147 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:25.585148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:25.585149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:25.585335 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:25.585572 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:25.585573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:25.585573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:25.585634 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:25.585870 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:25.585871 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:25.585915 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.586047 (+ 132us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.586135 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260807 22:45:25.586236 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.580305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:51033 (local address 127.16.98.190:42895)
0807 22:45:25.580350 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.580352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.580366 (+ 14us) server_negotiation.cc:407] Connection header received
0807 22:45:25.580471 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.580472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.580484 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.580536 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:25.580991 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.581693 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.582457 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.582633 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.582693 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.583722 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:25.583723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:25.583723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:25.583730 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:25.585060 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.585424 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.585425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.585425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.585503 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.585698 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.585698 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.585699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.585806 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:25.585857 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.586103 (+ 246us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.586174 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260807 22:45:25.588058 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 22:45:25 dist-test-slave-rgb1 krb5kdc[18501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 22:45:25 dist-test-slave-rgb1 krb5kdc[18501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:25.614094 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.605970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.16.98.190:42895)
0807 22:45:25.606084 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.606086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.606097 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:45:25.606135 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.606137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.606151 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.606227 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:25.606877 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.607461 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.608252 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.608440 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.608527 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.611403 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:25.611404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:25.611404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:25.611413 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:25.613059 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.613393 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.613393 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.613394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.613445 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.613618 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.613619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.613619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.613728 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:25.613814 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.613911 (+ 97us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.613978 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 22:45:25.616132 18554 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:45:25.617851 18554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:25.624974 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.620320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:44073 (local address 127.0.0.1:34624)
0807 22:45:25.620537 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.620602 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.620688 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.621238 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.621241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.621243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:25.621341 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.621343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.622698 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.622699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.623767 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.623769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.624724 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.624726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.624782 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.624785 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.624848 (+ 63us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.624879 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":71,"threads_started":1}
I20260807 22:45:25.624997 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.620837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.16.98.131:44073)
0807 22:45:25.621018 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.621022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.621031 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:45:25.621092 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.621094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.621112 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.621221 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:25.621442 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.622607 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.623834 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.624638 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.624715 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.624768 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.624862 (+ 94us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.624904 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20260807 22:45:25.625101 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.620614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.16.98.129:42677)
0807 22:45:25.621101 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.621105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.621115 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:45:25.621167 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.621169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.621186 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.621443 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:25.621502 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.622292 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.623788 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.624388 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.624478 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.624549 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.624943 (+ 394us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.624987 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":126,"threads_started":1}
I20260807 22:45:25.624881 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.620320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:42677 (local address 127.0.0.1:43606)
0807 22:45:25.620811 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.620822 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.620867 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.621296 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.621297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.621300 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:25.621359 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.621362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.622378 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.622380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.623695 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.623698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.624501 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.624502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.624624 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.624638 (+ 14us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.624721 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.624779 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":48,"thread_start_us":82,"threads_started":1}
I20260807 22:45:25.625511 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.620839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.16.98.130:35979)
0807 22:45:25.621432 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.621436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.621446 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:45:25.621495 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.621497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.621516 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.621595 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:25.621705 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.622805 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.623903 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.624653 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.624715 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.624769 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.625360 (+ 591us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.625409 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":94,"threads_started":1}
I20260807 22:45:25.625626 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.620593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:35979 (local address 127.0.0.1:36310)
0807 22:45:25.620826 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.620837 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.620879 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.621603 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.621603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.621605 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:25.621640 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.621642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.622891 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.622892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.623815 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.623818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.624805 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.624806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.624883 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.624887 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.624938 (+ 51us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.624983 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":80,"threads_started":1}
I20260807 22:45:25.625990 18991 tablet_service.cc:1467] Processing CreateTablet for tablet 95409de2fe264170b75bd0f895de30ba (DEFAULT_TABLE table=test-table [id=bb009a1600e04a3996e0842e41b06c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:25.626289 18843 tablet_service.cc:1467] Processing CreateTablet for tablet 95409de2fe264170b75bd0f895de30ba (DEFAULT_TABLE table=test-table [id=bb009a1600e04a3996e0842e41b06c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:25.626401 18991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95409de2fe264170b75bd0f895de30ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:25.626698 18843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95409de2fe264170b75bd0f895de30ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:25.629175 18699 tablet_service.cc:1467] Processing CreateTablet for tablet 95409de2fe264170b75bd0f895de30ba (DEFAULT_TABLE table=test-table [id=bb009a1600e04a3996e0842e41b06c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:25.629496 18699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95409de2fe264170b75bd0f895de30ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:25.630002 19085 tablet_bootstrap.cc:492] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499: Bootstrap starting.
I20260807 22:45:25.630311 19086 tablet_bootstrap.cc:492] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663: Bootstrap starting.
I20260807 22:45:25.630892 19086 tablet_bootstrap.cc:654] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:25.631064 19085 tablet_bootstrap.cc:654] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:25.631266 19086 log.cc:824] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:25.631428 19085 log.cc:824] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:25.632179 19086 tablet_bootstrap.cc:492] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663: No bootstrap required, opened a new log
I20260807 22:45:25.632164 19085 tablet_bootstrap.cc:492] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499: No bootstrap required, opened a new log
I20260807 22:45:25.632251 19086 ts_tablet_manager.cc:1397] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 22:45:25.632249 19085 ts_tablet_manager.cc:1397] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:45:25.633497 19089 tablet_bootstrap.cc:492] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf: Bootstrap starting.
I20260807 22:45:25.634274 19089 tablet_bootstrap.cc:654] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:25.634181 19085 raft_consensus.cc:348] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } }
I20260807 22:45:25.634346 19085 raft_consensus.cc:374] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:25.634232 19086 raft_consensus.cc:348] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } }
I20260807 22:45:25.634385 19085 raft_consensus.cc:729] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48670158931f461baaaf58a56e241499, State: Initialized, Role: FOLLOWER
I20260807 22:45:25.634393 19086 raft_consensus.cc:374] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:25.634433 19086 raft_consensus.cc:729] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7483aaa4b0c41ac9e67ebbf320f5663, State: Initialized, Role: FOLLOWER
I20260807 22:45:25.634539 19086 consensus_queue.cc:260] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } }
I20260807 22:45:25.634562 19085 consensus_queue.cc:260] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } }
I20260807 22:45:25.634635 19089 log.cc:824] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:25.634871 19085 ts_tablet_manager.cc:1428] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 22:45:25.634881 19086 ts_tablet_manager.cc:1428] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:45:25.634928 18913 heartbeater.cc:503] Master 127.16.98.190:42895 was elected leader, sending a full tablet report...
I20260807 22:45:25.634996 19060 heartbeater.cc:503] Master 127.16.98.190:42895 was elected leader, sending a full tablet report...
I20260807 22:45:25.635802 19089 tablet_bootstrap.cc:492] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf: No bootstrap required, opened a new log
I20260807 22:45:25.636018 19089 ts_tablet_manager.cc:1397] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:45:25.636139 19092 raft_consensus.cc:482] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:25.636230 19092 raft_consensus.cc:504] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } }
I20260807 22:45:25.637205 19092 leader_election.cc:288] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48670158931f461baaaf58a56e241499 (127.16.98.131:44073), 6b1d94c7563d45eebd68ffa45fa74baf (127.16.98.129:42677)
I20260807 22:45:25.638347 19089 raft_consensus.cc:348] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } }
I20260807 22:45:25.638547 19089 raft_consensus.cc:374] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:25.638597 19089 raft_consensus.cc:729] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1d94c7563d45eebd68ffa45fa74baf, State: Initialized, Role: FOLLOWER
I20260807 22:45:25.638734 19089 consensus_queue.cc:260] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } }
I20260807 22:45:25.639041 19089 ts_tablet_manager.cc:1428] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:45:25.639063 18766 heartbeater.cc:503] Master 127.16.98.190:42895 was elected leader, sending a full tablet report...
I20260807 22:45:25.639837 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.636741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:42677 (local address 127.16.98.130:32769)
0807 22:45:25.636837 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.636849 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.636893 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.637183 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.637185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.637187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:25.637273 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.637275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.638037 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.638038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.638819 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.638822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.639552 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.639554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.639642 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.639646 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.639704 (+ 58us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.639748 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260807 22:45:25.639880 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.636987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:32769 (local address 127.16.98.129:42677)
0807 22:45:25.637055 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.637057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.637064 (+ 7us) server_negotiation.cc:407] Connection header received
0807 22:45:25.637087 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.637087 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.637096 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.637165 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:25.637361 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.637950 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.638908 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.639467 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.639555 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.639565 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.639765 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.639817 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 22:45:25.640136 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95409de2fe264170b75bd0f895de30ba" candidate_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" is_pre_election: true
I20260807 22:45:25.640300 18719 raft_consensus.cc:2454] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7483aaa4b0c41ac9e67ebbf320f5663 in term 0.
W20260807 22:45:25.640337 18914 tablet.cc:2369] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:45:25.640607 18796 leader_election.cc:302] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b1d94c7563d45eebd68ffa45fa74baf, a7483aaa4b0c41ac9e67ebbf320f5663; no voters:
I20260807 22:45:25.640748 19092 raft_consensus.cc:2781] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:25.640812 19092 raft_consensus.cc:482] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:25.640841 19092 raft_consensus.cc:3037] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:25.641566 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.636930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:53781 (local address 127.16.98.131:44073)
0807 22:45:25.636984 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.636986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.637404 (+ 418us) server_negotiation.cc:407] Connection header received
0807 22:45:25.637460 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.637461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.637471 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.637541 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:25.637669 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.638362 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.640379 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.641051 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.641169 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.641180 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.641438 (+ 258us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.641483 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 22:45:25.641871 19092 raft_consensus.cc:504] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } }
I20260807 22:45:25.642036 19092 leader_election.cc:288] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [CANDIDATE]: Term 1 election: Requested vote from peers 48670158931f461baaaf58a56e241499 (127.16.98.131:44073), 6b1d94c7563d45eebd68ffa45fa74baf (127.16.98.129:42677)
I20260807 22:45:25.642107 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.636741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:44073 (local address 127.16.98.130:53781)
0807 22:45:25.637345 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.637356 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.637409 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.637546 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.637547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.637548 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:25.637607 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.637609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.638467 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.638468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.640215 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.640218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.641217 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.641219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.641298 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.641302 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.641420 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.641469 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":10,"thread_start_us":85,"threads_started":1}
I20260807 22:45:25.642196 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95409de2fe264170b75bd0f895de30ba" candidate_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1d94c7563d45eebd68ffa45fa74baf"
I20260807 22:45:25.642283 18719 raft_consensus.cc:3037] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:25.642581 19013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95409de2fe264170b75bd0f895de30ba" candidate_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48670158931f461baaaf58a56e241499" is_pre_election: true
I20260807 22:45:25.642589 19012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95409de2fe264170b75bd0f895de30ba" candidate_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48670158931f461baaaf58a56e241499"
I20260807 22:45:25.642750 19013 raft_consensus.cc:2454] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7483aaa4b0c41ac9e67ebbf320f5663 in term 0.
I20260807 22:45:25.643448 18719 raft_consensus.cc:2454] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7483aaa4b0c41ac9e67ebbf320f5663 in term 1.
I20260807 22:45:25.643647 18796 leader_election.cc:302] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b1d94c7563d45eebd68ffa45fa74baf, a7483aaa4b0c41ac9e67ebbf320f5663; no voters: 48670158931f461baaaf58a56e241499
I20260807 22:45:25.643762 19092 raft_consensus.cc:2781] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:25.643916 19092 raft_consensus.cc:686] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 LEADER]: Becoming Leader. State: Replica: a7483aaa4b0c41ac9e67ebbf320f5663, State: Running, Role: LEADER
I20260807 22:45:25.644012 19092 consensus_queue.cc:237] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } }
I20260807 22:45:25.644876 18554 catalog_manager.cc:5495] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7483aaa4b0c41ac9e67ebbf320f5663 (127.16.98.130). New cstate: current_term: 1 leader_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } health_report { overall_health: UNKNOWN } } }
Aug 07 22:45:25 dist-test-slave-rgb1 krb5kdc[18501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142724, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:25.679176 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.670029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:42895 (local address 127.0.0.1:56842)
0807 22:45:25.670287 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.670301 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.670387 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.670662 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.670663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.671071 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:25.671141 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.671144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.671931 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.671932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.672587 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.672589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.672973 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.672974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.673031 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.673640 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:25.673645 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:25.675512 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:25.677748 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:25.677760 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:25.677762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:25.678185 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:25.678418 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:25.678419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:25.678419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:25.678479 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:25.678795 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:25.678802 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:25.678869 (+ 67us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.678945 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.679089 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":184,"threads_started":1}
I20260807 22:45:25.679311 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.670130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.16.98.190:42895)
0807 22:45:25.670256 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.670258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.670474 (+ 216us) server_negotiation.cc:407] Connection header received
0807 22:45:25.670525 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.670526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.670542 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.670612 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:25.671337 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.671838 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.672692 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.672897 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.672973 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.675744 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:25.675747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:25.675747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:25.675755 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:25.677576 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.678273 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.678274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.678274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.678331 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:25.678540 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:25.678541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:25.678542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:25.678701 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:25.678758 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.679006 (+ 248us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.679162 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":80}
I20260807 22:45:25.681586 18554 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:56842:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:45:25.681782 18554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:25.683866 18843 tablet_service.cc:1467] Processing CreateTablet for tablet baff0c6c024a4b64804e87c3daff91bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9fd4876119c490bbba8fc15a814a06f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:45:25.683869 18991 tablet_service.cc:1467] Processing CreateTablet for tablet baff0c6c024a4b64804e87c3daff91bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9fd4876119c490bbba8fc15a814a06f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:45:25.684043 18991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baff0c6c024a4b64804e87c3daff91bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:25.684096 18843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baff0c6c024a4b64804e87c3daff91bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:25.685770 19086 tablet_bootstrap.cc:492] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663: Bootstrap starting.
I20260807 22:45:25.685920 18699 tablet_service.cc:1467] Processing CreateTablet for tablet baff0c6c024a4b64804e87c3daff91bc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9fd4876119c490bbba8fc15a814a06f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:45:25.686098 18699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baff0c6c024a4b64804e87c3daff91bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:25.686286 19086 tablet_bootstrap.cc:654] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:25.687299 19086 tablet_bootstrap.cc:492] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663: No bootstrap required, opened a new log
I20260807 22:45:25.687357 19086 ts_tablet_manager.cc:1397] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 22:45:25.687526 19086 raft_consensus.cc:348] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:25.687629 19086 raft_consensus.cc:374] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:25.687654 19086 raft_consensus.cc:729] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7483aaa4b0c41ac9e67ebbf320f5663, State: Initialized, Role: FOLLOWER
I20260807 22:45:25.687711 19086 consensus_queue.cc:260] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:25.687817 19086 ts_tablet_manager.cc:1428] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:25.687901 19085 tablet_bootstrap.cc:492] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499: Bootstrap starting.
I20260807 22:45:25.687891 19092 tablet_replica.cc:439] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } }
I20260807 22:45:25.687999 19092 tablet_replica.cc:442] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:25.687976 19089 tablet_bootstrap.cc:492] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf: Bootstrap starting.
I20260807 22:45:25.688498 19085 tablet_bootstrap.cc:654] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:25.688753 19089 tablet_bootstrap.cc:654] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:25.689420 19085 tablet_bootstrap.cc:492] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499: No bootstrap required, opened a new log
I20260807 22:45:25.689489 19085 ts_tablet_manager.cc:1397] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260807 22:45:25.689561 19089 tablet_bootstrap.cc:492] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf: No bootstrap required, opened a new log
I20260807 22:45:25.689620 19089 ts_tablet_manager.cc:1397] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 22:45:25.689664 19085 raft_consensus.cc:348] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:25.689771 19085 raft_consensus.cc:374] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:25.689802 19085 raft_consensus.cc:729] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48670158931f461baaaf58a56e241499, State: Initialized, Role: FOLLOWER
I20260807 22:45:25.689805 19089 raft_consensus.cc:348] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:25.689893 19089 raft_consensus.cc:374] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:25.689860 19085 consensus_queue.cc:260] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:25.689930 19089 raft_consensus.cc:729] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1d94c7563d45eebd68ffa45fa74baf, State: Initialized, Role: FOLLOWER
I20260807 22:45:25.689981 19085 ts_tablet_manager.cc:1428] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260807 22:45:25.690001 19089 consensus_queue.cc:260] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:25.690186 19091 tablet_replica.cc:439] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } }
I20260807 22:45:25.690199 19095 tablet_replica.cc:439] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } }
I20260807 22:45:25.690291 19095 tablet_replica.cc:442] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:25.690286 19091 tablet_replica.cc:442] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:25.690505 19089 ts_tablet_manager.cc:1428] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260807 22:45:25.707825 18767 tablet.cc:2369] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:45:25.824893 19061 tablet.cc:2369] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:45:25.915896 19091 raft_consensus.cc:482] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:25.916028 19091 raft_consensus.cc:504] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:25.916445 19091 leader_election.cc:288] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b1d94c7563d45eebd68ffa45fa74baf (127.16.98.129:42677), a7483aaa4b0c41ac9e67ebbf320f5663 (127.16.98.130:35979)
I20260807 22:45:25.919376 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.916833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:44679 (local address 127.16.98.129:42677)
0807 22:45:25.916880 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.916882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.916890 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:45:25.916917 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.916918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.916932 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.916989 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:25.917196 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.917702 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.918642 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.919002 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.919090 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.919101 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.919264 (+ 163us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.919305 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 22:45:25.919376 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.916684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:42677 (local address 127.16.98.131:44679)
0807 22:45:25.916754 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.916769 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.916829 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.917042 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.917043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.917045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:25.917114 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.917115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.917791 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.917792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.918545 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.918547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.919090 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.919091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.919186 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.919189 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.919259 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.919300 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260807 22:45:25.919452 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.916786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:49521 (local address 127.16.98.130:35979)
0807 22:45:25.916839 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.916842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.917005 (+ 163us) server_negotiation.cc:407] Connection header received
0807 22:45:25.917039 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.917041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.917051 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.917126 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:25.917286 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.917758 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.918704 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.919066 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.919150 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.919159 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.919333 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.919375 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 22:45:25.919519 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.916703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:35979 (local address 127.16.98.131:49521)
0807 22:45:25.916872 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.916881 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.916949 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.917130 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.917131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.917133 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:25.917214 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.917215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.917843 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.917845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.918608 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.918610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.919187 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.919188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.919255 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.919258 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.919327 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.919368 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":103,"threads_started":1}
I20260807 22:45:25.919868 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baff0c6c024a4b64804e87c3daff91bc" candidate_uuid: "48670158931f461baaaf58a56e241499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" is_pre_election: true
I20260807 22:45:25.920006 18861 raft_consensus.cc:2454] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48670158931f461baaaf58a56e241499 in term 0.
I20260807 22:45:25.919934 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baff0c6c024a4b64804e87c3daff91bc" candidate_uuid: "48670158931f461baaaf58a56e241499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" is_pre_election: true
I20260807 22:45:25.920120 18719 raft_consensus.cc:2454] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48670158931f461baaaf58a56e241499 in term 0.
I20260807 22:45:25.920233 18940 leader_election.cc:302] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48670158931f461baaaf58a56e241499, a7483aaa4b0c41ac9e67ebbf320f5663; no voters:
I20260807 22:45:25.920441 19091 raft_consensus.cc:2781] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:25.920506 19091 raft_consensus.cc:482] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:25.920542 19091 raft_consensus.cc:3037] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:25.921336 19091 raft_consensus.cc:504] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:25.921506 19091 leader_election.cc:288] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [CANDIDATE]: Term 1 election: Requested vote from peers 6b1d94c7563d45eebd68ffa45fa74baf (127.16.98.129:42677), a7483aaa4b0c41ac9e67ebbf320f5663 (127.16.98.130:35979)
I20260807 22:45:25.921712 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baff0c6c024a4b64804e87c3daff91bc" candidate_uuid: "48670158931f461baaaf58a56e241499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663"
I20260807 22:45:25.921736 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baff0c6c024a4b64804e87c3daff91bc" candidate_uuid: "48670158931f461baaaf58a56e241499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1d94c7563d45eebd68ffa45fa74baf"
I20260807 22:45:25.921810 18861 raft_consensus.cc:3037] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:25.921829 18719 raft_consensus.cc:3037] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:25.922531 18719 raft_consensus.cc:2454] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48670158931f461baaaf58a56e241499 in term 1.
I20260807 22:45:25.922744 18861 raft_consensus.cc:2454] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48670158931f461baaaf58a56e241499 in term 1.
I20260807 22:45:25.922997 18943 leader_election.cc:302] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48670158931f461baaaf58a56e241499, 6b1d94c7563d45eebd68ffa45fa74baf; no voters:
I20260807 22:45:25.923506 19091 raft_consensus.cc:2781] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:25.924058 19091 raft_consensus.cc:686] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [term 1 LEADER]: Becoming Leader. State: Replica: 48670158931f461baaaf58a56e241499, State: Running, Role: LEADER
I20260807 22:45:25.924207 19091 consensus_queue.cc:237] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:25.924794 19108 tablet_replica.cc:439] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499: TxnStatusTablet state changed. Reason: New leader 48670158931f461baaaf58a56e241499. Latest consensus state: current_term: 1 leader_uuid: "48670158931f461baaaf58a56e241499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } }
I20260807 22:45:25.924903 19108 tablet_replica.cc:442] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:45:25.925088 19110 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:45:25.925237 19110 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:45:25.925355 18554 catalog_manager.cc:5495] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48670158931f461baaaf58a56e241499 (127.16.98.131). New cstate: current_term: 1 leader_uuid: "48670158931f461baaaf58a56e241499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } health_report { overall_health: HEALTHY } } }
I20260807 22:45:25.978075 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.974374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:44073 (local address 127.0.0.1:34630)
0807 22:45:25.974488 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.974501 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.974589 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.974860 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.974861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.974862 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:45:25.974947 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.974950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.975949 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.975951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.976925 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.976930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.977428 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.977430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.977503 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.977531 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:45:25.977871 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:45:25.977882 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.977934 (+ 52us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.977992 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 22:45:25.978076 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.974484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34630 (local address 127.16.98.131:44073)
0807 22:45:25.974581 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.974584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.974596 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:45:25.974716 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.974717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.974751 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.974834 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:25.975043 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.975647 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.977042 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.977326 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.977416 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.977649 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:25.977770 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:25.977898 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.977955 (+ 57us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.977987 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 22:45:25.980134 18861 raft_consensus.cc:1261] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Refusing update from remote peer 48670158931f461baaaf58a56e241499: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:25.980204 18719 raft_consensus.cc:1261] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 FOLLOWER]: Refusing update from remote peer 48670158931f461baaaf58a56e241499: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:25.980458 19091 consensus_queue.cc:1035] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:25.980651 19108 consensus_queue.cc:1035] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:25.981781 19092 tablet_replica.cc:439] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663: TxnStatusTablet state changed. Reason: New leader 48670158931f461baaaf58a56e241499. Latest consensus state: current_term: 1 leader_uuid: "48670158931f461baaaf58a56e241499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } }
I20260807 22:45:25.981886 19092 tablet_replica.cc:442] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:25.981820 19095 tablet_replica.cc:439] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf: TxnStatusTablet state changed. Reason: New leader 48670158931f461baaaf58a56e241499. Latest consensus state: current_term: 1 leader_uuid: "48670158931f461baaaf58a56e241499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } }
I20260807 22:45:25.981922 19095 tablet_replica.cc:442] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:25.983276 19091 tablet_replica.cc:439] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48670158931f461baaaf58a56e241499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } }
I20260807 22:45:25.983374 19091 tablet_replica.cc:442] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:45:25.983462 19095 tablet_replica.cc:439] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48670158931f461baaaf58a56e241499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } }
I20260807 22:45:25.983564 19095 tablet_replica.cc:442] T baff0c6c024a4b64804e87c3daff91bc P 6b1d94c7563d45eebd68ffa45fa74baf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:25.983623 19114 mvcc.cc:205] Tried to move back new op lower bound from 7316040605611487232 to 7316040605386268672. Current Snapshot: MvccSnapshot[applied={T|T < 7316040605611487232}]
I20260807 22:45:25.984453 19092 tablet_replica.cc:439] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48670158931f461baaaf58a56e241499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } }
I20260807 22:45:25.984543 19092 tablet_replica.cc:442] T baff0c6c024a4b64804e87c3daff91bc P a7483aaa4b0c41ac9e67ebbf320f5663: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:25.983289 19108 tablet_replica.cc:439] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48670158931f461baaaf58a56e241499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } } }
I20260807 22:45:25.985265 19108 tablet_replica.cc:442] T baff0c6c024a4b64804e87c3daff91bc P 48670158931f461baaaf58a56e241499: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:45:25.985385 19115 mvcc.cc:205] Tried to move back new op lower bound from 7316040605611487232 to 7316040605386268672. Current Snapshot: MvccSnapshot[applied={T|T < 7316040605611487232 or (T in {7316040605611487232})}]
I20260807 22:45:25.988137 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.985640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36314 (local address 127.16.98.130:35979)
0807 22:45:25.985683 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.985685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.985708 (+ 23us) server_negotiation.cc:407] Connection header received
0807 22:45:25.985795 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.985796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.985814 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.985902 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:25.986058 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.986540 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.987311 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.987529 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.987606 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.987794 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:25.987903 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:25.987990 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.988033 (+ 43us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.988056 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260807 22:45:25.994336 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.991311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:56311 (local address 127.16.98.131:44073)
0807 22:45:25.991372 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:25.991374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:25.991426 (+ 52us) server_negotiation.cc:407] Connection header received
0807 22:45:25.991492 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:25.991493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:25.991505 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:25.991555 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:25.991719 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.992292 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.993412 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.993763 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.993823 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.994102 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:25.994134 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:25.994177 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:25.994262 (+ 85us) server_negotiation.cc:299] Negotiation successful
0807 22:45:25.994278 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260807 22:45:25.994614 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:25.991169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:44073 (local address 127.16.98.130:56311)
0807 22:45:25.991355 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:25.991367 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:25.991429 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:25.991581 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:25.991582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:25.991584 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:45:25.991648 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.991649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.992392 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.992393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.993313 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:25.993317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:25.993853 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:25.993854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:25.993974 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:25.993993 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:45:25.994200 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:45:25.994202 (+ 2us) client_negotiation.cc:769] Sending connection context
0807 22:45:25.994261 (+ 59us) client_negotiation.cc:240] Negotiation successful
0807 22:45:25.994312 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
I20260807 22:45:25.997499 18719 raft_consensus.cc:1261] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 FOLLOWER]: Refusing update from remote peer a7483aaa4b0c41ac9e67ebbf320f5663: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:25.997504 19012 raft_consensus.cc:3037] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:25.997766 19092 consensus_queue.cc:1035] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:25.998612 19012 raft_consensus.cc:1261] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Refusing update from remote peer a7483aaa4b0c41ac9e67ebbf320f5663: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:25.998858 19092 consensus_queue.cc:1035] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:26.001560 19113 mvcc.cc:205] Tried to move back new op lower bound from 7316040605683773440 to 7316040604238479360. Current Snapshot: MvccSnapshot[applied={T|T < 7316040605683773440}]
I20260807 22:45:26.009543 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:26.006063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:35979 (local address 127.16.98.131:39023)
0807 22:45:26.006144 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:26.006157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:26.006210 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:26.006343 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:26.006344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:26.006345 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:45:26.006836 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:26.006839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:26.007777 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:26.007778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:26.008620 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:26.008622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:26.009086 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:26.009087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:26.009157 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:26.009176 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:45:26.009371 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:45:26.009373 (+ 2us) client_negotiation.cc:769] Sending connection context
0807 22:45:26.009420 (+ 47us) client_negotiation.cc:240] Negotiation successful
0807 22:45:26.009470 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260807 22:45:26.009543 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:26.006104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:39023 (local address 127.16.98.130:35979)
0807 22:45:26.006147 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:26.006150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:26.006203 (+ 53us) server_negotiation.cc:407] Connection header received
0807 22:45:26.006265 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:26.006266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:26.006277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:26.006342 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:26.006931 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:26.007657 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:26.008716 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:26.009008 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:26.009089 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:26.009272 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:26.009309 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:26.009370 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:26.009431 (+ 61us) server_negotiation.cc:299] Negotiation successful
0807 22:45:26.009456 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 22:45:26.019929 18552 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:26.020098 18552 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:26.021025 18552 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386: Sending DeleteTablet for 3 replicas of tablet 95409de2fe264170b75bd0f895de30ba
I20260807 22:45:26.021415 18699 tablet_service.cc:1514] Processing DeleteTablet for tablet 95409de2fe264170b75bd0f895de30ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:26 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:43606
I20260807 22:45:26.021577 18991 tablet_service.cc:1514] Processing DeleteTablet for tablet 95409de2fe264170b75bd0f895de30ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:26 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:34624
I20260807 22:45:26.021715 18843 tablet_service.cc:1514] Processing DeleteTablet for tablet 95409de2fe264170b75bd0f895de30ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:26 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:36310
I20260807 22:45:26.022044 18552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:45:26.022207 18552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:26.022133 19134 tablet_replica.cc:330] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663: stopping tablet replica
I20260807 22:45:26.022482 19134 raft_consensus.cc:2229] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:45:26.022665 19134 raft_consensus.cc:2258] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:26.023253 19134 ts_tablet_manager.cc:1905] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:26.023438 18699 tablet_service.cc:1467] Processing CreateTablet for tablet 4132f83939c04dc0be9b7fb814b4ac46 (DEFAULT_TABLE table=test-table [id=6e09d360221f4594bee90e16d9bd6778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:26.023591 18699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4132f83939c04dc0be9b7fb814b4ac46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:26.023746 19132 tablet_replica.cc:330] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf: stopping tablet replica
I20260807 22:45:26.024689 19132 raft_consensus.cc:2229] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:26.024837 19132 raft_consensus.cc:2258] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:26.025100 19089 tablet_bootstrap.cc:492] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf: Bootstrap starting.
I20260807 22:45:26.025507 19132 ts_tablet_manager.cc:1905] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:26.025542 19089 tablet_bootstrap.cc:654] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:26.025631 18843 tablet_service.cc:1467] Processing CreateTablet for tablet 4132f83939c04dc0be9b7fb814b4ac46 (DEFAULT_TABLE table=test-table [id=6e09d360221f4594bee90e16d9bd6778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:26.025744 18843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4132f83939c04dc0be9b7fb814b4ac46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:26.027092 19086 tablet_bootstrap.cc:492] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663: Bootstrap starting.
I20260807 22:45:26.027266 18991 tablet_service.cc:1467] Processing CreateTablet for tablet 4132f83939c04dc0be9b7fb814b4ac46 (DEFAULT_TABLE table=test-table [id=6e09d360221f4594bee90e16d9bd6778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:26.027405 18991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4132f83939c04dc0be9b7fb814b4ac46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:26.027575 19086 tablet_bootstrap.cc:654] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:26.028326 19134 ts_tablet_manager.cc:1918] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:45:26.028460 19134 log.cc:1196] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/95409de2fe264170b75bd0f895de30ba
I20260807 22:45:26.029179 19132 ts_tablet_manager.cc:1918] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:45:26.029248 19085 tablet_bootstrap.cc:492] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499: Bootstrap starting.
I20260807 22:45:26.029633 19134 ts_tablet_manager.cc:1939] T 95409de2fe264170b75bd0f895de30ba P a7483aaa4b0c41ac9e67ebbf320f5663: Deleting consensus metadata
I20260807 22:45:26.029700 19085 tablet_bootstrap.cc:654] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:26.030249 19132 log.cc:1196] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/wal/wals/95409de2fe264170b75bd0f895de30ba
I20260807 22:45:26.030230 18532 catalog_manager.cc:4841] TS a7483aaa4b0c41ac9e67ebbf320f5663 (127.16.98.130:35979): tablet 95409de2fe264170b75bd0f895de30ba (table test-table [id=bb009a1600e04a3996e0842e41b06c21]) successfully deleted
I20260807 22:45:26.031123 19085 tablet_bootstrap.cc:492] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499: No bootstrap required, opened a new log
I20260807 22:45:26.031186 19085 ts_tablet_manager.cc:1397] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260807 22:45:26.031342 19085 raft_consensus.cc:348] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:26.031423 19085 raft_consensus.cc:374] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:26.031446 19085 raft_consensus.cc:729] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48670158931f461baaaf58a56e241499, State: Initialized, Role: FOLLOWER
I20260807 22:45:26.031490 19085 consensus_queue.cc:260] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:26.031589 19085 ts_tablet_manager.cc:1428] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260807 22:45:26.031749 19132 ts_tablet_manager.cc:1939] T 95409de2fe264170b75bd0f895de30ba P 6b1d94c7563d45eebd68ffa45fa74baf: Deleting consensus metadata
I20260807 22:45:26.031776 19133 tablet_replica.cc:330] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499: stopping tablet replica
I20260807 22:45:26.031913 19133 raft_consensus.cc:2229] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:26.031993 19133 raft_consensus.cc:2258] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:26.032686 18536 catalog_manager.cc:4841] TS 6b1d94c7563d45eebd68ffa45fa74baf (127.16.98.129:42677): tablet 95409de2fe264170b75bd0f895de30ba (table test-table [id=bb009a1600e04a3996e0842e41b06c21]) successfully deleted
I20260807 22:45:26.033131 19089 tablet_bootstrap.cc:492] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf: No bootstrap required, opened a new log
I20260807 22:45:26.033187 19089 ts_tablet_manager.cc:1397] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260807 22:45:26.033349 19089 raft_consensus.cc:348] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:26.033434 19089 raft_consensus.cc:374] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:26.033457 19089 raft_consensus.cc:729] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1d94c7563d45eebd68ffa45fa74baf, State: Initialized, Role: FOLLOWER
I20260807 22:45:26.033439 19133 ts_tablet_manager.cc:1905] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:26.033509 19089 consensus_queue.cc:260] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:26.033615 19089 ts_tablet_manager.cc:1428] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:26.035449 19133 ts_tablet_manager.cc:1918] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:45:26.035602 19133 log.cc:1196] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/95409de2fe264170b75bd0f895de30ba
I20260807 22:45:26.035975 19133 ts_tablet_manager.cc:1939] T 95409de2fe264170b75bd0f895de30ba P 48670158931f461baaaf58a56e241499: Deleting consensus metadata
I20260807 22:45:26.036470 18532 catalog_manager.cc:4841] TS 48670158931f461baaaf58a56e241499 (127.16.98.131:44073): tablet 95409de2fe264170b75bd0f895de30ba (table test-table [id=bb009a1600e04a3996e0842e41b06c21]) successfully deleted
I20260807 22:45:26.037070 19086 tablet_bootstrap.cc:492] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663: No bootstrap required, opened a new log
I20260807 22:45:26.037143 19086 ts_tablet_manager.cc:1397] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260807 22:45:26.037328 19086 raft_consensus.cc:348] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:26.037433 19086 raft_consensus.cc:374] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:26.037459 19086 raft_consensus.cc:729] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7483aaa4b0c41ac9e67ebbf320f5663, State: Initialized, Role: FOLLOWER
I20260807 22:45:26.037514 19086 consensus_queue.cc:260] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:26.037622 19086 ts_tablet_manager.cc:1428] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260807 22:45:26.143980 19108 raft_consensus.cc:482] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:26.144129 19108 raft_consensus.cc:504] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:26.144405 19108 leader_election.cc:288] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b1d94c7563d45eebd68ffa45fa74baf (127.16.98.129:42677), a7483aaa4b0c41ac9e67ebbf320f5663 (127.16.98.130:35979)
I20260807 22:45:26.144735 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4132f83939c04dc0be9b7fb814b4ac46" candidate_uuid: "48670158931f461baaaf58a56e241499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" is_pre_election: true
I20260807 22:45:26.144750 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4132f83939c04dc0be9b7fb814b4ac46" candidate_uuid: "48670158931f461baaaf58a56e241499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" is_pre_election: true
I20260807 22:45:26.144842 18719 raft_consensus.cc:2454] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48670158931f461baaaf58a56e241499 in term 0.
I20260807 22:45:26.144889 18861 raft_consensus.cc:2454] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48670158931f461baaaf58a56e241499 in term 0.
I20260807 22:45:26.145094 18943 leader_election.cc:302] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48670158931f461baaaf58a56e241499, 6b1d94c7563d45eebd68ffa45fa74baf; no voters:
I20260807 22:45:26.145192 19108 raft_consensus.cc:2781] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:26.145234 19108 raft_consensus.cc:482] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:26.145256 19108 raft_consensus.cc:3037] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:26.146253 19108 raft_consensus.cc:504] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:26.146417 19108 leader_election.cc:288] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [CANDIDATE]: Term 1 election: Requested vote from peers 6b1d94c7563d45eebd68ffa45fa74baf (127.16.98.129:42677), a7483aaa4b0c41ac9e67ebbf320f5663 (127.16.98.130:35979)
I20260807 22:45:26.146613 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4132f83939c04dc0be9b7fb814b4ac46" candidate_uuid: "48670158931f461baaaf58a56e241499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663"
I20260807 22:45:26.146658 18719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4132f83939c04dc0be9b7fb814b4ac46" candidate_uuid: "48670158931f461baaaf58a56e241499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1d94c7563d45eebd68ffa45fa74baf"
I20260807 22:45:26.146715 18861 raft_consensus.cc:3037] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:26.146730 18719 raft_consensus.cc:3037] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:26.147464 18861 raft_consensus.cc:2454] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48670158931f461baaaf58a56e241499 in term 1.
I20260807 22:45:26.147650 18940 leader_election.cc:302] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48670158931f461baaaf58a56e241499, a7483aaa4b0c41ac9e67ebbf320f5663; no voters:
I20260807 22:45:26.147655 18719 raft_consensus.cc:2454] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48670158931f461baaaf58a56e241499 in term 1.
I20260807 22:45:26.147758 19108 raft_consensus.cc:2781] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:26.147806 19108 raft_consensus.cc:686] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [term 1 LEADER]: Becoming Leader. State: Replica: 48670158931f461baaaf58a56e241499, State: Running, Role: LEADER
I20260807 22:45:26.147864 19108 consensus_queue.cc:237] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:26.148638 18552 catalog_manager.cc:5495] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48670158931f461baaaf58a56e241499 (127.16.98.131). New cstate: current_term: 1 leader_uuid: "48670158931f461baaaf58a56e241499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } health_report { overall_health: HEALTHY } } }
I20260807 22:45:26.184895 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:26.182537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34642 (local address 127.16.98.131:44073)
0807 22:45:26.182595 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:26.182597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:26.182605 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:45:26.182629 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:26.182630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:26.182645 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:26.182696 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:26.182909 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:26.183387 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:26.184139 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:26.184347 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:26.184437 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:26.184600 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:26.184640 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:26.184708 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:26.184773 (+ 65us) server_negotiation.cc:299] Negotiation successful
0807 22:45:26.184794 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260807 22:45:26.189316 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:26.187255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:44073 (local address 127.16.98.131:52695)
0807 22:45:26.187325 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:26.187338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:26.187391 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:26.187506 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:26.187506 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:26.187507 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:45:26.187544 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:26.187545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:26.188091 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:26.188092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:26.188688 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:26.188690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:26.189003 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:26.189004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:26.189040 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:26.189044 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:45:26.189170 (+ 126us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:45:26.189172 (+ 2us) client_negotiation.cc:769] Sending connection context
0807 22:45:26.189207 (+ 35us) client_negotiation.cc:240] Negotiation successful
0807 22:45:26.189250 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":4}
I20260807 22:45:26.189380 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:26.187332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:52695 (local address 127.16.98.131:44073)
0807 22:45:26.187390 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:26.187391 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:26.187399 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:45:26.187444 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:26.187444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:26.187453 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:26.187497 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:26.187583 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:26.188018 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:26.188762 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:26.188936 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:26.189040 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:26.189094 (+ 54us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:26.189117 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:26.189153 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:26.189208 (+ 55us) server_negotiation.cc:299] Negotiation successful
0807 22:45:26.189232 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":38}
I20260807 22:45:26.191697 18719 raft_consensus.cc:1261] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 FOLLOWER]: Refusing update from remote peer 48670158931f461baaaf58a56e241499: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:26.191955 18861 raft_consensus.cc:1261] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Refusing update from remote peer 48670158931f461baaaf58a56e241499: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:26.192015 19091 consensus_queue.cc:1035] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:26.192288 19091 consensus_queue.cc:1035] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 22:45:26.197319 19113 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 22:45:27.721501 18553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:27.721617 18553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:27.722546 18553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386: Sending DeleteTablet for 3 replicas of tablet 4132f83939c04dc0be9b7fb814b4ac46
I20260807 22:45:27.722894 18843 tablet_service.cc:1514] Processing DeleteTablet for tablet 4132f83939c04dc0be9b7fb814b4ac46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:27 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:36310
I20260807 22:45:27.722966 18990 tablet_service.cc:1514] Processing DeleteTablet for tablet 4132f83939c04dc0be9b7fb814b4ac46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:27 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:34624
I20260807 22:45:27.723017 18699 tablet_service.cc:1514] Processing DeleteTablet for tablet 4132f83939c04dc0be9b7fb814b4ac46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:27 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:43606
I20260807 22:45:27.723201 19164 tablet_replica.cc:330] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499: stopping tablet replica
I20260807 22:45:27.723272 19164 raft_consensus.cc:2229] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:45:27.723380 19164 raft_consensus.cc:2258] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:27.723660 19164 ts_tablet_manager.cc:1905] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:27.725473 19164 ts_tablet_manager.cc:1918] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 22:45:27.725558 19164 log.cc:1196] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/4132f83939c04dc0be9b7fb814b4ac46
I20260807 22:45:27.725812 19164 ts_tablet_manager.cc:1939] T 4132f83939c04dc0be9b7fb814b4ac46 P 48670158931f461baaaf58a56e241499: Deleting consensus metadata
I20260807 22:45:27.726619 18532 catalog_manager.cc:4841] TS 48670158931f461baaaf58a56e241499 (127.16.98.131:44073): tablet 4132f83939c04dc0be9b7fb814b4ac46 (table test-table [id=6e09d360221f4594bee90e16d9bd6778]) successfully deleted
I20260807 22:45:27.726655 19165 tablet_replica.cc:330] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf: stopping tablet replica
I20260807 22:45:27.726743 19165 raft_consensus.cc:2229] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:27.726801 19165 raft_consensus.cc:2258] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:27.727082 19165 ts_tablet_manager.cc:1905] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:27.728681 19163 tablet_replica.cc:330] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663: stopping tablet replica
I20260807 22:45:27.728770 19163 raft_consensus.cc:2229] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:27.728826 19163 raft_consensus.cc:2258] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:27.729090 19163 ts_tablet_manager.cc:1905] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:27.729085 19165 ts_tablet_manager.cc:1918] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 22:45:27.729162 19165 log.cc:1196] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/wal/wals/4132f83939c04dc0be9b7fb814b4ac46
I20260807 22:45:27.729429 19165 ts_tablet_manager.cc:1939] T 4132f83939c04dc0be9b7fb814b4ac46 P 6b1d94c7563d45eebd68ffa45fa74baf: Deleting consensus metadata
I20260807 22:45:27.729938 18536 catalog_manager.cc:4841] TS 6b1d94c7563d45eebd68ffa45fa74baf (127.16.98.129:42677): tablet 4132f83939c04dc0be9b7fb814b4ac46 (table test-table [id=6e09d360221f4594bee90e16d9bd6778]) successfully deleted
I20260807 22:45:27.730571 19163 ts_tablet_manager.cc:1918] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 22:45:27.730655 19163 log.cc:1196] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/4132f83939c04dc0be9b7fb814b4ac46
I20260807 22:45:27.730865 19163 ts_tablet_manager.cc:1939] T 4132f83939c04dc0be9b7fb814b4ac46 P a7483aaa4b0c41ac9e67ebbf320f5663: Deleting consensus metadata
I20260807 22:45:27.731251 18532 catalog_manager.cc:4841] TS a7483aaa4b0c41ac9e67ebbf320f5663 (127.16.98.130:35979): tablet 4132f83939c04dc0be9b7fb814b4ac46 (table test-table [id=6e09d360221f4594bee90e16d9bd6778]) successfully deleted
Aug 07 22:45:27 dist-test-slave-rgb1 krb5kdc[18501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 22:45:27 dist-test-slave-rgb1 krb5kdc[18501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:27.756152 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:27.743042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.16.98.190:42895)
0807 22:45:27.743218 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:27.743221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:27.743230 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:45:27.743260 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:27.743260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:27.743271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:27.743336 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:27.744065 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.744815 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.745729 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.745957 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.746021 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:27.752765 (+ 6744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:27.752767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:27.752768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:27.752776 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:27.754465 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:27.754931 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:27.754932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:27.754933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:27.755013 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:27.755366 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:27.755368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:27.755368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:27.755598 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:27.755738 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:27.755937 (+ 199us) server_negotiation.cc:299] Negotiation successful
0807 22:45:27.756044 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":105,"threads_started":1}
I20260807 22:45:27.758235 18553 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:45:27.758622 18553 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:27.760351 18843 tablet_service.cc:1467] Processing CreateTablet for tablet 588385010ff44784b105afad3eacff78 (DEFAULT_TABLE table=test-table [id=066cf4d73a7c4d6aad4b085e6af0efed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:27.760351 18699 tablet_service.cc:1467] Processing CreateTablet for tablet 588385010ff44784b105afad3eacff78 (DEFAULT_TABLE table=test-table [id=066cf4d73a7c4d6aad4b085e6af0efed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:27.760392 18990 tablet_service.cc:1467] Processing CreateTablet for tablet 588385010ff44784b105afad3eacff78 (DEFAULT_TABLE table=test-table [id=066cf4d73a7c4d6aad4b085e6af0efed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:27.760517 18843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 588385010ff44784b105afad3eacff78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:27.760526 18699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 588385010ff44784b105afad3eacff78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:27.760565 18990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 588385010ff44784b105afad3eacff78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:27.762038 19175 tablet_bootstrap.cc:492] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf: Bootstrap starting.
I20260807 22:45:27.762202 19176 tablet_bootstrap.cc:492] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663: Bootstrap starting.
I20260807 22:45:27.762295 19174 tablet_bootstrap.cc:492] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499: Bootstrap starting.
I20260807 22:45:27.762727 19175 tablet_bootstrap.cc:654] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:27.762727 19174 tablet_bootstrap.cc:654] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:27.763358 19176 tablet_bootstrap.cc:654] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:27.763579 19175 tablet_bootstrap.cc:492] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf: No bootstrap required, opened a new log
I20260807 22:45:27.763651 19175 ts_tablet_manager.cc:1397] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 22:45:27.763732 19174 tablet_bootstrap.cc:492] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499: No bootstrap required, opened a new log
I20260807 22:45:27.763787 19174 ts_tablet_manager.cc:1397] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 22:45:27.763854 19175 raft_consensus.cc:348] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:27.763962 19175 raft_consensus.cc:374] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:27.763919 19174 raft_consensus.cc:348] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:27.763998 19175 raft_consensus.cc:729] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1d94c7563d45eebd68ffa45fa74baf, State: Initialized, Role: FOLLOWER
I20260807 22:45:27.764005 19174 raft_consensus.cc:374] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:27.764030 19174 raft_consensus.cc:729] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48670158931f461baaaf58a56e241499, State: Initialized, Role: FOLLOWER
I20260807 22:45:27.764063 19175 consensus_queue.cc:260] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:27.764278 19175 ts_tablet_manager.cc:1428] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:27.764108 19174 consensus_queue.cc:260] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:27.764472 19174 ts_tablet_manager.cc:1428] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 22:45:27.765188 19176 tablet_bootstrap.cc:492] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663: No bootstrap required, opened a new log
I20260807 22:45:27.765269 19176 ts_tablet_manager.cc:1397] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260807 22:45:27.765406 19176 raft_consensus.cc:348] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:27.765534 19176 raft_consensus.cc:374] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:27.765558 19176 raft_consensus.cc:729] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7483aaa4b0c41ac9e67ebbf320f5663, State: Initialized, Role: FOLLOWER
I20260807 22:45:27.765626 19176 consensus_queue.cc:260] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:27.765831 19176 ts_tablet_manager.cc:1428] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:27.869832 19180 raft_consensus.cc:482] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:27.870003 19180 raft_consensus.cc:504] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:27.870456 19180 leader_election.cc:288] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7483aaa4b0c41ac9e67ebbf320f5663 (127.16.98.130:35979), 48670158931f461baaaf58a56e241499 (127.16.98.131:44073)
I20260807 22:45:27.874771 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:27.870896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:42953 (local address 127.16.98.130:35979)
0807 22:45:27.871252 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:27.871254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:27.871266 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:45:27.871299 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:27.871300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:27.871313 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:27.871386 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:27.871608 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.872408 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.873716 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.874372 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.874433 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:27.874441 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:27.874657 (+ 216us) server_negotiation.cc:299] Negotiation successful
0807 22:45:27.874697 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":186,"threads_started":1}
I20260807 22:45:27.874768 19182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:27.870748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:35979 (local address 127.16.98.129:42953)
0807 22:45:27.871052 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:27.871067 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:27.871155 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:27.871400 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:27.871401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:27.871404 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:27.871491 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:27.871494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.872505 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.872506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:27.873612 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:27.873615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.874466 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.874467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:27.874556 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:27.874562 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 22:45:27.874627 (+ 65us) client_negotiation.cc:240] Negotiation successful
0807 22:45:27.874672 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":200,"threads_started":1}
I20260807 22:45:27.875177 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588385010ff44784b105afad3eacff78" candidate_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" is_pre_election: true
I20260807 22:45:27.875317 18861 raft_consensus.cc:2454] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1d94c7563d45eebd68ffa45fa74baf in term 0.
I20260807 22:45:27.875355 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:27.871076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:44073 (local address 127.16.98.129:43607)
0807 22:45:27.871229 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:27.871241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:27.871301 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:27.871599 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:27.871600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:27.871603 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:27.872044 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:27.872113 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.873146 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.873147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:27.874218 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:27.874221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.875097 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.875099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:27.875187 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:27.875191 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:27.875236 (+ 45us) client_negotiation.cc:240] Negotiation successful
0807 22:45:27.875264 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260807 22:45:27.875411 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:27.871182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:43607 (local address 127.16.98.131:44073)
0807 22:45:27.871371 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:27.871374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:27.871384 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:45:27.871411 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:27.871411 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:27.871424 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:27.871519 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:27.872235 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.873048 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.874322 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.874950 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.875029 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:27.875039 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:27.875268 (+ 229us) server_negotiation.cc:299] Negotiation successful
0807 22:45:27.875330 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":121,"threads_started":1}
I20260807 22:45:27.875548 18646 leader_election.cc:302] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b1d94c7563d45eebd68ffa45fa74baf, a7483aaa4b0c41ac9e67ebbf320f5663; no voters:
I20260807 22:45:27.875702 19180 raft_consensus.cc:2781] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:27.875779 19180 raft_consensus.cc:482] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:27.875808 19180 raft_consensus.cc:3037] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:27.876048 19012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588385010ff44784b105afad3eacff78" candidate_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48670158931f461baaaf58a56e241499" is_pre_election: true
I20260807 22:45:27.876144 19012 raft_consensus.cc:2454] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1d94c7563d45eebd68ffa45fa74baf in term 0.
I20260807 22:45:27.876917 19180 raft_consensus.cc:504] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:27.877065 19180 leader_election.cc:288] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [CANDIDATE]: Term 1 election: Requested vote from peers a7483aaa4b0c41ac9e67ebbf320f5663 (127.16.98.130:35979), 48670158931f461baaaf58a56e241499 (127.16.98.131:44073)
I20260807 22:45:27.877238 19012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588385010ff44784b105afad3eacff78" candidate_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48670158931f461baaaf58a56e241499"
I20260807 22:45:27.877240 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588385010ff44784b105afad3eacff78" candidate_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663"
I20260807 22:45:27.877333 19012 raft_consensus.cc:3037] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:27.877333 18861 raft_consensus.cc:3037] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:27.878247 19012 raft_consensus.cc:2454] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1d94c7563d45eebd68ffa45fa74baf in term 1.
I20260807 22:45:27.878280 18861 raft_consensus.cc:2454] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1d94c7563d45eebd68ffa45fa74baf in term 1.
I20260807 22:45:27.878468 18646 leader_election.cc:302] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48670158931f461baaaf58a56e241499, 6b1d94c7563d45eebd68ffa45fa74baf; no voters:
I20260807 22:45:27.878598 19180 raft_consensus.cc:2781] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:27.878773 19180 raft_consensus.cc:686] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 LEADER]: Becoming Leader. State: Replica: 6b1d94c7563d45eebd68ffa45fa74baf, State: Running, Role: LEADER
I20260807 22:45:27.878836 19180 consensus_queue.cc:237] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:27.879626 18552 catalog_manager.cc:5495] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b1d94c7563d45eebd68ffa45fa74baf (127.16.98.129). New cstate: current_term: 1 leader_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } health_report { overall_health: UNKNOWN } } }
I20260807 22:45:27.917418 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:27.914459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.16.98.129:42677)
0807 22:45:27.914627 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:27.914629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:27.914635 (+ 6us) server_negotiation.cc:407] Connection header received
0807 22:45:27.914677 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:27.914678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:27.914701 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:27.914896 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:27.915155 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.915744 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.916531 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.916767 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.916849 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:27.917030 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:27.917143 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:27.917232 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:27.917279 (+ 47us) server_negotiation.cc:299] Negotiation successful
0807 22:45:27.917301 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
I20260807 22:45:27.924702 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:27.920799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:59111 (local address 127.16.98.131:44073)
0807 22:45:27.920887 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:27.920889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:27.920916 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:45:27.920941 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:27.920941 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:27.920951 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:27.921083 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:27.921813 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.922670 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.923642 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.923838 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.923897 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:27.924249 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:27.924315 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:27.924417 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:27.924542 (+ 125us) server_negotiation.cc:299] Negotiation successful
0807 22:45:27.924629 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 22:45:27.924736 19193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:27.920503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:44073 (local address 127.16.98.129:59111)
0807 22:45:27.920747 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:27.920763 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:27.920826 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:27.921363 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:27.921365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:27.921366 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:45:27.921435 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:27.921437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.922775 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.922776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:27.923552 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:27.923555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.923960 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.923962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:27.924022 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:27.924043 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:45:27.924439 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:45:27.924441 (+ 2us) client_negotiation.cc:769] Sending connection context
0807 22:45:27.924497 (+ 56us) client_negotiation.cc:240] Negotiation successful
0807 22:45:27.924615 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":134,"threads_started":1}
I20260807 22:45:27.927420 19012 raft_consensus.cc:1261] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Refusing update from remote peer 6b1d94c7563d45eebd68ffa45fa74baf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:27.927516 18861 raft_consensus.cc:1261] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Refusing update from remote peer 6b1d94c7563d45eebd68ffa45fa74baf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:27.927800 19180 consensus_queue.cc:1035] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [LEADER]: Connected to new peer: Peer: permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:27.927985 19187 consensus_queue.cc:1035] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:27.936733 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:27.934085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:57791 (local address 127.16.98.129:42677)
0807 22:45:27.934128 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:27.934130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:27.934172 (+ 42us) server_negotiation.cc:407] Connection header received
0807 22:45:27.934237 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:27.934237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:27.934246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:27.934293 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:27.934389 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.934888 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.935763 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.936093 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.936179 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:27.936387 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:27.936431 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:27.936496 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:27.936599 (+ 103us) server_negotiation.cc:299] Negotiation successful
0807 22:45:27.936626 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260807 22:45:27.936787 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:27.933937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:42677 (local address 127.16.98.131:57791)
0807 22:45:27.934116 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:27.934128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:27.934186 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:27.934294 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:27.934295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:27.934296 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:45:27.934343 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:27.934344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.934962 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.934963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:27.935650 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:27.935653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:27.936228 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:27.936229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:27.936297 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:27.936302 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:45:27.936500 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:45:27.936502 (+ 2us) client_negotiation.cc:769] Sending connection context
0807 22:45:27.936594 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 22:45:27.936650 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":112,"threads_started":1}
I20260807 22:45:27.948465 18552 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:27.948688 18552 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:27.949672 18552 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386: Sending DeleteTablet for 3 replicas of tablet 588385010ff44784b105afad3eacff78
I20260807 22:45:27.949962 18699 tablet_service.cc:1514] Processing DeleteTablet for tablet 588385010ff44784b105afad3eacff78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:27 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:43606
I20260807 22:45:27.950071 19165 tablet_replica.cc:330] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf: stopping tablet replica
I20260807 22:45:27.950093 18990 tablet_service.cc:1514] Processing DeleteTablet for tablet 588385010ff44784b105afad3eacff78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:27 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:34624
I20260807 22:45:27.950127 19165 raft_consensus.cc:2229] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:45:27.950222 19165 raft_consensus.cc:2258] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:27.950225 19164 tablet_replica.cc:330] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499: stopping tablet replica
I20260807 22:45:27.950295 19164 raft_consensus.cc:2229] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:27.950340 19164 raft_consensus.cc:2258] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:27.950331 18843 tablet_service.cc:1514] Processing DeleteTablet for tablet 588385010ff44784b105afad3eacff78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:27 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:36310
I20260807 22:45:27.950449 19163 tablet_replica.cc:330] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663: stopping tablet replica
I20260807 22:45:27.950513 19163 raft_consensus.cc:2229] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:27.950573 19163 raft_consensus.cc:2258] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:27.950860 19165 ts_tablet_manager.cc:1905] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:27.950963 18552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:45:27.951121 18552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:27.951593 19164 ts_tablet_manager.cc:1905] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:27.952682 19163 ts_tablet_manager.cc:1905] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:27.953438 19164 ts_tablet_manager.cc:1918] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:45:27.953508 19164 log.cc:1196] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/588385010ff44784b105afad3eacff78
I20260807 22:45:27.953747 19164 ts_tablet_manager.cc:1939] T 588385010ff44784b105afad3eacff78 P 48670158931f461baaaf58a56e241499: Deleting consensus metadata
I20260807 22:45:27.954741 18699 tablet_service.cc:1467] Processing CreateTablet for tablet eebdf805a21141238edefae5b0ccd432 (DEFAULT_TABLE table=test-table [id=5670086160be41158297e234c8452f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:27.954905 18990 tablet_service.cc:1467] Processing CreateTablet for tablet eebdf805a21141238edefae5b0ccd432 (DEFAULT_TABLE table=test-table [id=5670086160be41158297e234c8452f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:27.955013 18990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebdf805a21141238edefae5b0ccd432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:27.956516 19165 ts_tablet_manager.cc:1918] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:45:27.956640 19165 log.cc:1196] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-0/wal/wals/588385010ff44784b105afad3eacff78
I20260807 22:45:27.956789 19174 tablet_bootstrap.cc:492] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499: Bootstrap starting.
I20260807 22:45:27.956934 19165 ts_tablet_manager.cc:1939] T 588385010ff44784b105afad3eacff78 P 6b1d94c7563d45eebd68ffa45fa74baf: Deleting consensus metadata
I20260807 22:45:27.957142 19174 tablet_bootstrap.cc:654] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:27.957603 18532 catalog_manager.cc:4841] TS 48670158931f461baaaf58a56e241499 (127.16.98.131:44073): tablet 588385010ff44784b105afad3eacff78 (table test-table [id=066cf4d73a7c4d6aad4b085e6af0efed]) successfully deleted
I20260807 22:45:27.957830 18699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebdf805a21141238edefae5b0ccd432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:27.957907 19174 tablet_bootstrap.cc:492] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499: No bootstrap required, opened a new log
I20260807 22:45:27.957962 19174 ts_tablet_manager.cc:1397] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:27.958106 18536 catalog_manager.cc:4841] TS 6b1d94c7563d45eebd68ffa45fa74baf (127.16.98.129:42677): tablet 588385010ff44784b105afad3eacff78 (table test-table [id=066cf4d73a7c4d6aad4b085e6af0efed]) successfully deleted
I20260807 22:45:27.958141 19174 raft_consensus.cc:348] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:27.958338 19174 raft_consensus.cc:374] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:27.958379 19174 raft_consensus.cc:729] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48670158931f461baaaf58a56e241499, State: Initialized, Role: FOLLOWER
I20260807 22:45:27.958541 19174 consensus_queue.cc:260] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:27.958709 19174 ts_tablet_manager.cc:1428] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 22:45:27.959267 19175 tablet_bootstrap.cc:492] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf: Bootstrap starting.
I20260807 22:45:27.959590 18843 tablet_service.cc:1467] Processing CreateTablet for tablet eebdf805a21141238edefae5b0ccd432 (DEFAULT_TABLE table=test-table [id=5670086160be41158297e234c8452f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:27.959772 19175 tablet_bootstrap.cc:654] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:27.959787 19163 ts_tablet_manager.cc:1918] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:45:27.959842 19163 log.cc:1196] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/588385010ff44784b105afad3eacff78
I20260807 22:45:27.960110 19163 ts_tablet_manager.cc:1939] T 588385010ff44784b105afad3eacff78 P a7483aaa4b0c41ac9e67ebbf320f5663: Deleting consensus metadata
I20260807 22:45:27.960363 18843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebdf805a21141238edefae5b0ccd432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:27.960703 18532 catalog_manager.cc:4841] TS a7483aaa4b0c41ac9e67ebbf320f5663 (127.16.98.130:35979): tablet 588385010ff44784b105afad3eacff78 (table test-table [id=066cf4d73a7c4d6aad4b085e6af0efed]) successfully deleted
I20260807 22:45:27.961756 19176 tablet_bootstrap.cc:492] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663: Bootstrap starting.
I20260807 22:45:27.962131 19176 tablet_bootstrap.cc:654] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:27.962661 19175 tablet_bootstrap.cc:492] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf: No bootstrap required, opened a new log
I20260807 22:45:27.962733 19175 ts_tablet_manager.cc:1397] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260807 22:45:27.963004 19175 raft_consensus.cc:348] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:27.963091 19176 tablet_bootstrap.cc:492] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663: No bootstrap required, opened a new log
I20260807 22:45:27.963104 19175 raft_consensus.cc:374] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:27.963128 19175 raft_consensus.cc:729] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1d94c7563d45eebd68ffa45fa74baf, State: Initialized, Role: FOLLOWER
I20260807 22:45:27.963141 19176 ts_tablet_manager.cc:1397] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:27.963176 19175 consensus_queue.cc:260] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:27.963296 19175 ts_tablet_manager.cc:1428] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:27.963263 19176 raft_consensus.cc:348] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:27.963338 19176 raft_consensus.cc:374] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:27.963361 19176 raft_consensus.cc:729] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7483aaa4b0c41ac9e67ebbf320f5663, State: Initialized, Role: FOLLOWER
I20260807 22:45:27.963413 19176 consensus_queue.cc:260] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:27.963518 19176 ts_tablet_manager.cc:1428] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260807 22:45:28.000970 19180 raft_consensus.cc:482] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:28.001089 19180 raft_consensus.cc:504] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:28.001271 19180 leader_election.cc:288] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7483aaa4b0c41ac9e67ebbf320f5663 (127.16.98.130:35979), 48670158931f461baaaf58a56e241499 (127.16.98.131:44073)
I20260807 22:45:28.001566 19012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebdf805a21141238edefae5b0ccd432" candidate_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48670158931f461baaaf58a56e241499" is_pre_election: true
I20260807 22:45:28.001570 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebdf805a21141238edefae5b0ccd432" candidate_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" is_pre_election: true
I20260807 22:45:28.001684 18861 raft_consensus.cc:2454] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1d94c7563d45eebd68ffa45fa74baf in term 0.
I20260807 22:45:28.001684 19012 raft_consensus.cc:2454] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1d94c7563d45eebd68ffa45fa74baf in term 0.
I20260807 22:45:28.001871 18646 leader_election.cc:302] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48670158931f461baaaf58a56e241499, 6b1d94c7563d45eebd68ffa45fa74baf; no voters:
I20260807 22:45:28.001993 19180 raft_consensus.cc:2781] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:28.002050 19180 raft_consensus.cc:482] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:28.002080 19180 raft_consensus.cc:3037] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:28.002987 19180 raft_consensus.cc:504] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:28.003151 19180 leader_election.cc:288] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [CANDIDATE]: Term 1 election: Requested vote from peers a7483aaa4b0c41ac9e67ebbf320f5663 (127.16.98.130:35979), 48670158931f461baaaf58a56e241499 (127.16.98.131:44073)
I20260807 22:45:28.003376 19012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebdf805a21141238edefae5b0ccd432" candidate_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48670158931f461baaaf58a56e241499"
I20260807 22:45:28.003376 18861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eebdf805a21141238edefae5b0ccd432" candidate_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663"
I20260807 22:45:28.003495 19012 raft_consensus.cc:3037] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:28.003495 18861 raft_consensus.cc:3037] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:28.004277 19012 raft_consensus.cc:2454] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1d94c7563d45eebd68ffa45fa74baf in term 1.
I20260807 22:45:28.004349 18861 raft_consensus.cc:2454] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1d94c7563d45eebd68ffa45fa74baf in term 1.
I20260807 22:45:28.004592 18646 leader_election.cc:302] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48670158931f461baaaf58a56e241499, 6b1d94c7563d45eebd68ffa45fa74baf; no voters:
I20260807 22:45:28.004705 19180 raft_consensus.cc:2781] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:28.004778 19180 raft_consensus.cc:686] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [term 1 LEADER]: Becoming Leader. State: Replica: 6b1d94c7563d45eebd68ffa45fa74baf, State: Running, Role: LEADER
I20260807 22:45:28.004840 19180 consensus_queue.cc:237] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } }
I20260807 22:45:28.005491 18552 catalog_manager.cc:5495] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b1d94c7563d45eebd68ffa45fa74baf (127.16.98.129). New cstate: current_term: 1 leader_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1d94c7563d45eebd68ffa45fa74baf" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 } health_report { overall_health: UNKNOWN } } }
I20260807 22:45:28.023155 18861 raft_consensus.cc:1261] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Refusing update from remote peer 6b1d94c7563d45eebd68ffa45fa74baf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:28.023142 19012 raft_consensus.cc:1261] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Refusing update from remote peer 6b1d94c7563d45eebd68ffa45fa74baf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:28.023413 19180 consensus_queue.cc:1035] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7483aaa4b0c41ac9e67ebbf320f5663" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 35979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:28.023540 19198 consensus_queue.cc:1035] T eebdf805a21141238edefae5b0ccd432 P 6b1d94c7563d45eebd68ffa45fa74baf [LEADER]: Connected to new peer: Peer: permanent_uuid: "48670158931f461baaaf58a56e241499" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 22:45:28.028280 19147 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 22:45:29.551440 18554 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:29.551584 18554 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:29.552672 18554 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37c4d63f6a6a42dda3125546d1cba386: Sending DeleteTablet for 3 replicas of tablet eebdf805a21141238edefae5b0ccd432
I20260807 22:45:29.552968 18699 tablet_service.cc:1514] Processing DeleteTablet for tablet eebdf805a21141238edefae5b0ccd432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:29 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:43606
I20260807 22:45:29.553025 18990 tablet_service.cc:1514] Processing DeleteTablet for tablet eebdf805a21141238edefae5b0ccd432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:29 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:34624
I20260807 22:45:29.553040 18843 tablet_service.cc:1514] Processing DeleteTablet for tablet eebdf805a21141238edefae5b0ccd432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:29 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:36310
I20260807 22:45:29.553243 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 18629
I20260807 22:45:29.553294 19235 tablet_replica.cc:330] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663: stopping tablet replica
I20260807 22:45:29.554383 19235 raft_consensus.cc:2229] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:29.554451 19235 raft_consensus.cc:2258] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:29.554783 19235 ts_tablet_manager.cc:1905] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:29.555464 19233 tablet_replica.cc:330] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499: stopping tablet replica
I20260807 22:45:29.555547 19233 raft_consensus.cc:2229] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:29.555608 19233 raft_consensus.cc:2258] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:29.555917 19233 ts_tablet_manager.cc:1905] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:29.557189 19235 ts_tablet_manager.cc:1918] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 22:45:29.557281 19235 log.cc:1196] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/eebdf805a21141238edefae5b0ccd432
I20260807 22:45:29.557587 19235 ts_tablet_manager.cc:1939] T eebdf805a21141238edefae5b0ccd432 P a7483aaa4b0c41ac9e67ebbf320f5663: Deleting consensus metadata
I20260807 22:45:29.557812 19233 ts_tablet_manager.cc:1918] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 22:45:29.557894 19233 log.cc:1196] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/eebdf805a21141238edefae5b0ccd432
I20260807 22:45:29.558043 18532 catalog_manager.cc:4841] TS a7483aaa4b0c41ac9e67ebbf320f5663 (127.16.98.130:35979): tablet eebdf805a21141238edefae5b0ccd432 (table test-table [id=5670086160be41158297e234c8452f2d]) successfully deleted
I20260807 22:45:29.558161 19233 ts_tablet_manager.cc:1939] T eebdf805a21141238edefae5b0ccd432 P 48670158931f461baaaf58a56e241499: Deleting consensus metadata
I20260807 22:45:29.558634 18532 catalog_manager.cc:4841] TS 48670158931f461baaaf58a56e241499 (127.16.98.131:44073): tablet eebdf805a21141238edefae5b0ccd432 (table test-table [id=5670086160be41158297e234c8452f2d]) successfully deleted
W20260807 22:45:29.563203 18536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:42677 (error 108)
I20260807 22:45:29.563594 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 18776
I20260807 22:45:29.567782 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:29.563646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:42677 (local address 127.0.0.1:43622)
0807 22:45:29.567627 (+ 3981us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:29.567709 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3943,"thread_start_us":91,"threads_started":1}
W20260807 22:45:29.567924 18536 catalog_manager.cc:4584] TS 6b1d94c7563d45eebd68ffa45fa74baf (127.16.98.129:42677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eebdf805a21141238edefae5b0ccd432: Network error: Client connection negotiation failed: client connection to 127.16.98.129:42677: connect: Connection refused (error 111)
I20260807 22:45:29.574975 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 18923
I20260807 22:45:29.582617 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 18515
2026-08-07T22:45:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6057 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:45:29 dist-test-slave-rgb1 krb5kdc[19240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:45:29 dist-test-slave-rgb1 krb5kdc[19240](info): set up 2 sockets
Aug 07 22:45:29 dist-test-slave-rgb1 krb5kdc[19240](info): commencing operation
krb5kdc: starting...
W20260807 22:45:30.685022 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:45:30 dist-test-slave-rgb1 krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:45:30.833601 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:38311
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:38311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:30.923789 19253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:30.924018 19253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:30.924043 19253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:30.925305 19253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:30.925354 19253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:30.925369 19253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:30.925382 19253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:30.925395 19253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:30.926962 19253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:38311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:38311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:30.927233 19253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:30.927506 19253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:30.933179 19261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:30.933230 19253 server_base.cc:1034] running on GCE node
W20260807 22:45:30.933413 19258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:30.933243 19259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:30.933776 19253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:30.934083 19253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:30.935241 19253 hybrid_clock.cc:648] HybridClock initialized: now 1786142730935213 us; error 41 us; skew 500 ppm
Aug 07 22:45:30 dist-test-slave-rgb1 krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142730, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:30.938303 19253 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:45:30.938868 19253 webserver.cc:460] Webserver started at http://127.16.98.190:46847/ using document root <none> and password file <none>
I20260807 22:45:30.939184 19253 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:30.939275 19253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:30.939450 19253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:30.940774 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "c458be73888946d7946287d38a6c89e5"
format_stamp: "Formatted at 2026-08-07 22:45:30 on dist-test-slave-rgb1"
server_key: "297751e362b6ca72c67bd259bdd7d325"
server_key_iv: "9c31c24031a9d4c950f0bfe229d26227"
server_key_version: "encryptionkey@0"
I20260807 22:45:30.941187 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "c458be73888946d7946287d38a6c89e5"
format_stamp: "Formatted at 2026-08-07 22:45:30 on dist-test-slave-rgb1"
server_key: "297751e362b6ca72c67bd259bdd7d325"
server_key_iv: "9c31c24031a9d4c950f0bfe229d26227"
server_key_version: "encryptionkey@0"
I20260807 22:45:30.942822 19253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260807 22:45:30.943799 19268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:30.944130 19253 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:30.944301 19253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "c458be73888946d7946287d38a6c89e5"
format_stamp: "Formatted at 2026-08-07 22:45:30 on dist-test-slave-rgb1"
server_key: "297751e362b6ca72c67bd259bdd7d325"
server_key_iv: "9c31c24031a9d4c950f0bfe229d26227"
server_key_version: "encryptionkey@0"
I20260807 22:45:30.944397 19253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:30.958309 19253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:30.961169 19253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:30.961361 19253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:30.966703 19253 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:38311
I20260807 22:45:30.966848 19321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:38311 every 8 connection(s)
I20260807 22:45:30.967140 19253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:30.968250 19322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:30.970832 19322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5: Bootstrap starting.
I20260807 22:45:30.971108 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 19253
I20260807 22:45:30.971246 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:30.971473 16778 external_mini_cluster.cc:1428] Setting key 035d7bc9489ce058ec51f87397fdf90f
I20260807 22:45:30.971871 19322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:30.972270 19322 log.cc:824] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:30.973165 19322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5: No bootstrap required, opened a new log
I20260807 22:45:30.975600 19322 raft_consensus.cc:348] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c458be73888946d7946287d38a6c89e5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38311 } }
I20260807 22:45:30.975790 19322 raft_consensus.cc:374] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:30.975824 19322 raft_consensus.cc:729] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c458be73888946d7946287d38a6c89e5, State: Initialized, Role: FOLLOWER
I20260807 22:45:30.975988 19322 consensus_queue.cc:260] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c458be73888946d7946287d38a6c89e5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38311 } }
I20260807 22:45:30.976065 19322 raft_consensus.cc:388] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:30.976095 19322 raft_consensus.cc:482] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:30.976135 19322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:30.976854 19322 raft_consensus.cc:504] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c458be73888946d7946287d38a6c89e5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38311 } }
I20260807 22:45:30.976984 19322 leader_election.cc:302] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c458be73888946d7946287d38a6c89e5; no voters:
I20260807 22:45:30.977171 19322 leader_election.cc:288] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:30.977242 19327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:30.977432 19327 raft_consensus.cc:686] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [term 1 LEADER]: Becoming Leader. State: Replica: c458be73888946d7946287d38a6c89e5, State: Running, Role: LEADER
I20260807 22:45:30.977510 19322 sys_catalog.cc:564] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:30.977594 19327 consensus_queue.cc:237] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c458be73888946d7946287d38a6c89e5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38311 } }
I20260807 22:45:30.977938 19329 sys_catalog.cc:455] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c458be73888946d7946287d38a6c89e5. Latest consensus state: current_term: 1 leader_uuid: "c458be73888946d7946287d38a6c89e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c458be73888946d7946287d38a6c89e5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38311 } } }
I20260807 22:45:30.978094 19329 sys_catalog.cc:458] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:30.978722 19335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:30.978796 19330 sys_catalog.cc:455] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c458be73888946d7946287d38a6c89e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c458be73888946d7946287d38a6c89e5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38311 } } }
I20260807 22:45:30.978895 19330 sys_catalog.cc:458] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:30.979220 19335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 07 22:45:30 dist-test-slave-rgb1 krb5kdc[19240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:30.981725 19335 catalog_manager.cc:1347] Generated new cluster ID: bc205622940f477bb38ae07378a8df6d
I20260807 22:45:30.981788 19335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:45:30.984007 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:30.972226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60940 (local address 127.16.98.190:38311)
0807 22:45:30.972503 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:30.972511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:30.972540 (+ 29us) server_negotiation.cc:407] Connection header received
0807 22:45:30.973079 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:30.973086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:30.973137 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:30.973313 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:30.974129 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:30.975135 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:30.976219 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:30.976727 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:30.976820 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:30.980103 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:30.980108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:30.980109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:30.980123 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:30.982376 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:30.982792 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:30.982795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:30.982795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:30.982897 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:30.983117 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:30.983118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:30.983119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:30.983378 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:30.983560 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:30.983673 (+ 113us) server_negotiation.cc:299] Negotiation successful
0807 22:45:30.983819 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":95,"threads_started":1}
I20260807 22:45:31.001039 19335 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:31.001776 19335 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:31.003868 19335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c458be73888946d7946287d38a6c89e5: Generated new TSK 0
I20260807 22:45:31.004220 19335 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:45:31.040798 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:38311
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:31.129751 19349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:31.130015 19349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:31.130061 19349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:31.131305 19349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:31.131378 19349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:31.131429 19349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:45:31.133044 19349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:31.133332 19349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:31.133594 19349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 22:45:31.140216 19349 server_base.cc:1034] running on GCE node
W20260807 22:45:31.140149 19358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:31.140241 19355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:31.140149 19354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:31.140744 19349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:31.141040 19349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:31.142402 19349 hybrid_clock.cc:648] HybridClock initialized: now 1786142731142339 us; error 70 us; skew 500 ppm
Aug 07 22:45:31 dist-test-slave-rgb1 krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142731, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:31.146257 19349 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:45:31.146881 19349 webserver.cc:460] Webserver started at http://127.16.98.129:41735/ using document root <none> and password file <none>
I20260807 22:45:31.147182 19349 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:31.147266 19349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:31.147472 19349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:31.148865 19349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "487d312838f44d52b3ba756d21e98e7d"
format_stamp: "Formatted at 2026-08-07 22:45:31 on dist-test-slave-rgb1"
server_key: "07b1ebdf78583d6959a93d2d8a41fd65"
server_key_iv: "37c239053754547fa758032bf796a455"
server_key_version: "encryptionkey@0"
I20260807 22:45:31.149281 19349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "487d312838f44d52b3ba756d21e98e7d"
format_stamp: "Formatted at 2026-08-07 22:45:31 on dist-test-slave-rgb1"
server_key: "07b1ebdf78583d6959a93d2d8a41fd65"
server_key_iv: "37c239053754547fa758032bf796a455"
server_key_version: "encryptionkey@0"
I20260807 22:45:31.150930 19349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:45:31.151988 19365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:31.152274 19349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:31.152402 19349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "487d312838f44d52b3ba756d21e98e7d"
format_stamp: "Formatted at 2026-08-07 22:45:31 on dist-test-slave-rgb1"
server_key: "07b1ebdf78583d6959a93d2d8a41fd65"
server_key_iv: "37c239053754547fa758032bf796a455"
server_key_version: "encryptionkey@0"
I20260807 22:45:31.152529 19349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:31.163587 19349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:31.166464 19349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:31.166666 19349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:31.166954 19349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:31.167322 19349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:31.167387 19349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:31.167449 19349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:31.167481 19349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:31.175506 19349 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:41631
I20260807 22:45:31.175643 19484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:41631 every 8 connection(s)
I20260807 22:45:31.175969 19349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:45:31.177611 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 19349
I20260807 22:45:31.177896 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:45:31.178161 16778 external_mini_cluster.cc:1428] Setting key 2d9bc1f55272174373831707a06bd74f
Aug 07 22:45:31 dist-test-slave-rgb1 krb5kdc[19240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142731, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
I20260807 22:45:31.187988 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:31.176937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:57189 (local address 127.16.98.190:38311)
0807 22:45:31.177051 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:31.177053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:31.177332 (+ 279us) server_negotiation.cc:407] Connection header received
0807 22:45:31.177785 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:31.177789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:31.177813 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:31.177953 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:31.179335 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:31.180078 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:31.181397 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:31.181627 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:31.181719 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:31.184694 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:31.184695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:31.184695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:31.184704 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:31.186253 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:31.186742 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:31.186743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:31.186744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:31.186822 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:31.187088 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:31.187089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:31.187090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:31.187220 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:31.187427 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:31.187753 (+ 326us) server_negotiation.cc:299] Negotiation successful
0807 22:45:31.187876 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 22:45:31.188200 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:31.176878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:38311 (local address 127.16.98.129:57189)
0807 22:45:31.177059 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:31.177107 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:31.177605 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:31.178043 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:31.178047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:31.178866 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:31.179186 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:31.179194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:31.180257 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:31.180258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:31.181282 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:31.181285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:31.181736 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:31.181736 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:31.181872 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:31.182550 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:31.182559 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:31.184501 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:31.186379 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:31.186389 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:31.186392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:31.186647 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:31.186894 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:31.186895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:31.186895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:31.187014 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:31.187411 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:31.187421 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:31.187591 (+ 170us) client_negotiation.cc:769] Sending connection context
0807 22:45:31.187727 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 22:45:31.187919 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
I20260807 22:45:31.189152 19485 heartbeater.cc:347] Connected to a master server at 127.16.98.190:38311
I20260807 22:45:31.189327 19485 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:31.189604 19485 heartbeater.cc:511] Master 127.16.98.190:38311 requested a full tablet report, sending...
I20260807 22:45:31.190590 19286 ts_manager.cc:194] Registered new tserver with Master: 487d312838f44d52b3ba756d21e98e7d (127.16.98.129:41631)
I20260807 22:45:31.191521 19286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:57189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:45:31.213200 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:38311
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:31.306825 19490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:31.307094 19490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:31.307125 19490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:31.308637 19490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:31.308703 19490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:31.308737 19490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:45:31.310562 19490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:31.310896 19490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:31.311187 19490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:31.317076 19495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:31.317229 19497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:31.317094 19499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:31.317317 19490 server_base.cc:1034] running on GCE node
I20260807 22:45:31.317530 19490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:31.317752 19490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:31.318924 19490 hybrid_clock.cc:648] HybridClock initialized: now 1786142731318885 us; error 51 us; skew 500 ppm
Aug 07 22:45:31 dist-test-slave-rgb1 krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142731, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:31.322121 19490 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:45:31.322674 19490 webserver.cc:460] Webserver started at http://127.16.98.130:42615/ using document root <none> and password file <none>
I20260807 22:45:31.322956 19490 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:31.323020 19490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:31.323159 19490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:31.324337 19490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "358cbc6303f14c2d8198267b6be725d9"
format_stamp: "Formatted at 2026-08-07 22:45:31 on dist-test-slave-rgb1"
server_key: "6777575f8c9fa3f859cac1927cfc21a6"
server_key_iv: "0043e1b8dc428f6eddc7ba656c6e7a77"
server_key_version: "encryptionkey@0"
I20260807 22:45:31.324833 19490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "358cbc6303f14c2d8198267b6be725d9"
format_stamp: "Formatted at 2026-08-07 22:45:31 on dist-test-slave-rgb1"
server_key: "6777575f8c9fa3f859cac1927cfc21a6"
server_key_iv: "0043e1b8dc428f6eddc7ba656c6e7a77"
server_key_version: "encryptionkey@0"
I20260807 22:45:31.326553 19490 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260807 22:45:31.328125 19506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:31.328441 19490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260807 22:45:31.328621 19490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "358cbc6303f14c2d8198267b6be725d9"
format_stamp: "Formatted at 2026-08-07 22:45:31 on dist-test-slave-rgb1"
server_key: "6777575f8c9fa3f859cac1927cfc21a6"
server_key_iv: "0043e1b8dc428f6eddc7ba656c6e7a77"
server_key_version: "encryptionkey@0"
I20260807 22:45:31.328727 19490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:31.337934 19490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:31.341048 19490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:31.341409 19490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:31.341992 19490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:31.342439 19490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:31.342489 19490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:31.342514 19490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:31.342527 19490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:31.349633 19490 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:43957
I20260807 22:45:31.349689 19623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:43957 every 8 connection(s)
I20260807 22:45:31.350083 19490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
Aug 07 22:45:31 dist-test-slave-rgb1 krb5kdc[19240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142731, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:31.360069 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 19490
I20260807 22:45:31.360174 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:45:31.360414 16778 external_mini_cluster.cc:1428] Setting key 4d5d7d75a6b589d273e0ebb856d60b8c
I20260807 22:45:31.362810 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:31.350979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:34647 (local address 127.16.98.190:38311)
0807 22:45:31.351115 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:31.351118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:31.351279 (+ 161us) server_negotiation.cc:407] Connection header received
0807 22:45:31.351855 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:31.351856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:31.351870 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:31.352006 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:31.353152 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:31.353681 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:31.354710 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:31.355056 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:31.355259 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:31.358716 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:31.358718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:31.358718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:31.358737 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:31.360836 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:31.361336 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:31.361337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:31.361338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:31.361410 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:31.361861 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:31.361862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:31.361862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:31.362019 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:31.362222 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:31.362607 (+ 385us) server_negotiation.cc:299] Negotiation successful
0807 22:45:31.362711 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 22:45:31.363042 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:31.350897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:38311 (local address 127.16.98.130:34647)
0807 22:45:31.351141 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:31.351191 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:31.351739 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:31.352088 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:31.352092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:31.352826 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:31.353038 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:31.353042 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:31.353800 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:31.353801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:31.354613 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:31.354615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:31.355306 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:31.355308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:31.355572 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:31.356409 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:31.356427 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:31.358577 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:31.360930 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:31.360944 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:31.360947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:31.361237 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:31.361482 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:31.361483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:31.361483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:31.361736 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:31.362210 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:31.362219 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:31.362433 (+ 214us) client_negotiation.cc:769] Sending connection context
0807 22:45:31.362566 (+ 133us) client_negotiation.cc:240] Negotiation successful
0807 22:45:31.362760 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":93,"threads_started":1}
I20260807 22:45:31.363616 19624 heartbeater.cc:347] Connected to a master server at 127.16.98.190:38311
I20260807 22:45:31.363737 19624 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:31.363967 19624 heartbeater.cc:511] Master 127.16.98.190:38311 requested a full tablet report, sending...
I20260807 22:45:31.364408 19286 ts_manager.cc:194] Registered new tserver with Master: 358cbc6303f14c2d8198267b6be725d9 (127.16.98.130:43957)
I20260807 22:45:31.364908 19286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:34647
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:45:31.405505 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:38311
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:31.502187 19629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:31.502447 19629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:31.502476 19629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:31.503759 19629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:31.503814 19629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:31.503887 19629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:45:31.505966 19629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:31.506287 19629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:31.506645 19629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:31.514153 19635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:31.514200 19637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:31.514585 19634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:31.516005 19629 server_base.cc:1034] running on GCE node
I20260807 22:45:31.516348 19629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:31.516757 19629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:31.517990 19629 hybrid_clock.cc:648] HybridClock initialized: now 1786142731517960 us; error 51 us; skew 500 ppm
Aug 07 22:45:31 dist-test-slave-rgb1 krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142731, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:31.521551 19629 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:45:31.522164 19629 webserver.cc:460] Webserver started at http://127.16.98.131:41137/ using document root <none> and password file <none>
I20260807 22:45:31.522429 19629 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:31.522502 19629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:31.522670 19629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:31.523885 19629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "8b71c2db0ffe4bc0bd62449cac003f71"
format_stamp: "Formatted at 2026-08-07 22:45:31 on dist-test-slave-rgb1"
server_key: "9a6af51fcb74a5ea3d116701e35ed3e6"
server_key_iv: "2f4fc1e11532a15d4ba1223ce1b14a71"
server_key_version: "encryptionkey@0"
I20260807 22:45:31.524299 19629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "8b71c2db0ffe4bc0bd62449cac003f71"
format_stamp: "Formatted at 2026-08-07 22:45:31 on dist-test-slave-rgb1"
server_key: "9a6af51fcb74a5ea3d116701e35ed3e6"
server_key_iv: "2f4fc1e11532a15d4ba1223ce1b14a71"
server_key_version: "encryptionkey@0"
I20260807 22:45:31.525946 19629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:45:31.527176 19644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:31.527617 19629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:31.527763 19629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "8b71c2db0ffe4bc0bd62449cac003f71"
format_stamp: "Formatted at 2026-08-07 22:45:31 on dist-test-slave-rgb1"
server_key: "9a6af51fcb74a5ea3d116701e35ed3e6"
server_key_iv: "2f4fc1e11532a15d4ba1223ce1b14a71"
server_key_version: "encryptionkey@0"
I20260807 22:45:31.527859 19629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:31.542757 19629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:31.545686 19629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:31.545825 19629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:31.546131 19629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:31.546505 19629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:31.546542 19629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:31.546566 19629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:31.546579 19629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:31.553135 19629 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:46421
I20260807 22:45:31.553184 19761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:46421 every 8 connection(s)
I20260807 22:45:31.553580 19629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
Aug 07 22:45:31 dist-test-slave-rgb1 krb5kdc[19240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142731, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:31.563882 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 19629
I20260807 22:45:31.564155 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:45:31.564795 16778 external_mini_cluster.cc:1428] Setting key b040df35e15e8fc0173b4d2bc974f9cc
I20260807 22:45:31.567687 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:31.554916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:49349 (local address 127.16.98.190:38311)
0807 22:45:31.555129 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:31.555131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:31.555604 (+ 473us) server_negotiation.cc:407] Connection header received
0807 22:45:31.556172 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:31.556173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:31.556186 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:31.556263 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:31.557560 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:31.558302 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:31.559437 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:31.559685 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:31.559758 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:31.562646 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:31.562648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:31.562648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:31.562657 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:31.565498 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:31.566268 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:31.566270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:31.566270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:31.566347 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:31.566891 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:31.566892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:31.566893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:31.567073 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:31.567151 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:31.567517 (+ 366us) server_negotiation.cc:299] Negotiation successful
0807 22:45:31.567602 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260807 22:45:31.567898 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:31.554886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:38311 (local address 127.16.98.131:49349)
0807 22:45:31.555356 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:31.555427 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:31.556016 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:31.556424 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:31.556429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:31.557180 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:31.557422 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:31.557428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:31.558427 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:31.558427 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:31.559322 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:31.559325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:31.559818 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:31.559819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:31.559943 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:31.560702 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:31.560711 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:31.562502 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:31.565636 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:31.565791 (+ 155us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:31.565798 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:31.566138 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:31.566563 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:31.566564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:31.566565 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:31.566733 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:31.567193 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:31.567204 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:31.567347 (+ 143us) client_negotiation.cc:769] Sending connection context
0807 22:45:31.567456 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 22:45:31.567654 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":285,"threads_started":1}
I20260807 22:45:31.568758 19762 heartbeater.cc:347] Connected to a master server at 127.16.98.190:38311
I20260807 22:45:31.568871 19762 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:31.569049 19762 heartbeater.cc:511] Master 127.16.98.190:38311 requested a full tablet report, sending...
I20260807 22:45:31.569573 19286 ts_manager.cc:194] Registered new tserver with Master: 8b71c2db0ffe4bc0bd62449cac003f71 (127.16.98.131:46421)
I20260807 22:45:31.569996 19286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:49349
I20260807 22:45:31.579118 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 22:45:31.594834 19325 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:31.593319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60944 (local address 127.16.98.190:38311)
0807 22:45:31.593403 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:31.593405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:31.593779 (+ 374us) server_negotiation.cc:407] Connection header received
0807 22:45:31.593834 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:31.593835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:31.593847 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:31.593927 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:31.594772 (+ 845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60944: BlockingRecv error: recv got EOF from 127.0.0.1:60944 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 22:45:31.596061 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 19349
I20260807 22:45:31.602378 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 19490
I20260807 22:45:31.609963 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 19629
I20260807 22:45:31.615643 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 19253
2026-08-07T22:45:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2033 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:45:31 dist-test-slave-rgb1 krb5kdc[19774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:45:31 dist-test-slave-rgb1 krb5kdc[19774](info): set up 2 sockets
Aug 07 22:45:31 dist-test-slave-rgb1 krb5kdc[19774](info): commencing operation
krb5kdc: starting...
W20260807 22:45:32.790869 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:45:32 dist-test-slave-rgb1 krb5kdc[19774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:45:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:45:32.949259 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:45111
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:34419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:45111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:33.059262 19788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:33.059646 19788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:33.059684 19788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:33.061380 19788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:33.061442 19788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:33.061460 19788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:33.061478 19788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:33.061493 19788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:33.063566 19788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:45111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:33.063916 19788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:33.064213 19788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:33.070494 19796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:33.070675 19788 server_base.cc:1034] running on GCE node
W20260807 22:45:33.070401 19793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:33.070925 19794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:33.071300 19788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:33.071671 19788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:33.072831 19788 hybrid_clock.cc:648] HybridClock initialized: now 1786142733072805 us; error 58 us; skew 500 ppm
Aug 07 22:45:33 dist-test-slave-rgb1 krb5kdc[19774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142733, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:33.082166 19788 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:45:33.082813 19788 webserver.cc:460] Webserver started at http://127.16.98.190:46179/ using document root <none> and password file <none>
I20260807 22:45:33.083102 19788 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:33.083179 19788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:33.083341 19788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:33.084507 19788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "7e34c9c476a24b0b99930e990c6f2451"
format_stamp: "Formatted at 2026-08-07 22:45:33 on dist-test-slave-rgb1"
server_key: "582a3599b9e5ef75cb773938506eac87"
server_key_iv: "d62c09c954c3121afc9ca449c8d03ccd"
server_key_version: "encryptionkey@0"
I20260807 22:45:33.084985 19788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "7e34c9c476a24b0b99930e990c6f2451"
format_stamp: "Formatted at 2026-08-07 22:45:33 on dist-test-slave-rgb1"
server_key: "582a3599b9e5ef75cb773938506eac87"
server_key_iv: "d62c09c954c3121afc9ca449c8d03ccd"
server_key_version: "encryptionkey@0"
I20260807 22:45:33.086547 19788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:33.087579 19803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:33.087786 19788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:33.087889 19788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "7e34c9c476a24b0b99930e990c6f2451"
format_stamp: "Formatted at 2026-08-07 22:45:33 on dist-test-slave-rgb1"
server_key: "582a3599b9e5ef75cb773938506eac87"
server_key_iv: "d62c09c954c3121afc9ca449c8d03ccd"
server_key_version: "encryptionkey@0"
I20260807 22:45:33.087952 19788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:33.108269 19788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:33.111879 19788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:33.112097 19788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:33.119267 19788 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:45111
I20260807 22:45:33.119381 19856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:45111 every 8 connection(s)
I20260807 22:45:33.120007 19788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:33.120908 19857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:33.124040 19857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451: Bootstrap starting.
I20260807 22:45:33.124790 19857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:33.125188 19857 log.cc:824] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:33.126151 19857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451: No bootstrap required, opened a new log
I20260807 22:45:33.127923 19857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e34c9c476a24b0b99930e990c6f2451" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 45111 } }
I20260807 22:45:33.128127 19857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:33.128176 19857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e34c9c476a24b0b99930e990c6f2451, State: Initialized, Role: FOLLOWER
I20260807 22:45:33.128401 19857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e34c9c476a24b0b99930e990c6f2451" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 45111 } }
I20260807 22:45:33.128511 19857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:33.128592 19857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:33.128651 19857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:33.128753 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 19788
I20260807 22:45:33.128876 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:33.129077 16778 external_mini_cluster.cc:1428] Setting key 72001fb393cfc55fe15d13127a4486ad
I20260807 22:45:33.130089 19857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e34c9c476a24b0b99930e990c6f2451" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 45111 } }
I20260807 22:45:33.130201 19857 leader_election.cc:302] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e34c9c476a24b0b99930e990c6f2451; no voters:
I20260807 22:45:33.130398 19857 leader_election.cc:288] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:33.130455 19862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:33.130672 19862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [term 1 LEADER]: Becoming Leader. State: Replica: 7e34c9c476a24b0b99930e990c6f2451, State: Running, Role: LEADER
I20260807 22:45:33.130761 19857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:33.130779 19862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e34c9c476a24b0b99930e990c6f2451" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 45111 } }
I20260807 22:45:33.131436 19863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e34c9c476a24b0b99930e990c6f2451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e34c9c476a24b0b99930e990c6f2451" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 45111 } } }
I20260807 22:45:33.131462 19864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e34c9c476a24b0b99930e990c6f2451. Latest consensus state: current_term: 1 leader_uuid: "7e34c9c476a24b0b99930e990c6f2451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e34c9c476a24b0b99930e990c6f2451" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 45111 } } }
I20260807 22:45:33.131618 19863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:33.131644 19864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:33.132329 19871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:33.132911 19871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:33.135046 19871 catalog_manager.cc:1347] Generated new cluster ID: 946a457d86344734ab7db6e730f0d7e5
I20260807 22:45:33.135103 19871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 22:45:33 dist-test-slave-rgb1 krb5kdc[19774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:33.141917 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:33.129649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.16.98.190:45111)
0807 22:45:33.129907 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:33.129917 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:33.129955 (+ 38us) server_negotiation.cc:407] Connection header received
0807 22:45:33.130079 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:33.130087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:33.130151 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:33.130326 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:33.131270 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:33.132804 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:33.133845 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:33.134491 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:33.134610 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:33.137676 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:33.137681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:33.137681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:33.137695 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:33.140049 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:33.140485 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:33.140489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:33.140489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:33.140646 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:33.140866 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:33.140867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:33.140867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:33.141232 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:33.141416 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:33.141494 (+ 78us) server_negotiation.cc:299] Negotiation successful
0807 22:45:33.141663 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":122,"threads_started":1}
I20260807 22:45:33.143492 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:45:33.143756 19871 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:33.144754 19871 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:33.152827 19871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e34c9c476a24b0b99930e990c6f2451: Generated new TSK 0
I20260807 22:45:33.153100 19871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:45:33.170084 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:33.164908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38540 (local address 127.16.98.190:45111)
0807 22:45:33.164982 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:33.164985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:33.165122 (+ 137us) server_negotiation.cc:407] Connection header received
0807 22:45:33.165188 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:33.165189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:33.165203 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:33.165296 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:33.165747 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:33.166671 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:33.167496 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:33.167922 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:33.168017 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:33.169343 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:33.169345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:33.169345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:33.169689 (+ 344us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:33.169716 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:33.169725 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:33.169807 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:33.169944 (+ 137us) server_negotiation.cc:299] Negotiation successful
0807 22:45:33.169981 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 22:45:33.172927 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 19788
2026-08-07T22:45:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1555 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:45:33 dist-test-slave-rgb1 krb5kdc[19890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:45:33 dist-test-slave-rgb1 krb5kdc[19890](info): set up 2 sockets
Aug 07 22:45:33 dist-test-slave-rgb1 krb5kdc[19890](info): commencing operation
krb5kdc: starting...
W20260807 22:45:34.348275 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:45:34 dist-test-slave-rgb1 krb5kdc[19890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:45:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:45:34.488785 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:40183
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:40183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:34.580626 19903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:34.580899 19903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:34.580930 19903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:34.582365 19903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:34.582423 19903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:34.582443 19903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:34.582460 19903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:34.582477 19903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:34.584282 19903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:40183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:40183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:34.584678 19903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:34.584932 19903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:34.590714 19911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:34.590801 19908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:34.590734 19909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:34.590994 19903 server_base.cc:1034] running on GCE node
I20260807 22:45:34.591348 19903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:34.591711 19903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:34.592935 19903 hybrid_clock.cc:648] HybridClock initialized: now 1786142734592896 us; error 53 us; skew 500 ppm
Aug 07 22:45:34 dist-test-slave-rgb1 krb5kdc[19890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142734, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:34.596213 19903 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:45:34.596824 19903 webserver.cc:460] Webserver started at http://127.16.98.190:42001/ using document root <none> and password file <none>
I20260807 22:45:34.597091 19903 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:34.597147 19903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:34.597334 19903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:34.598572 19903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "e5a19ec0ee44430dae086a88fbd33245"
format_stamp: "Formatted at 2026-08-07 22:45:34 on dist-test-slave-rgb1"
server_key: "cdb7289a4dafc94df2797c9b04b14321"
server_key_iv: "ab6c517245639840a061412efc2b4701"
server_key_version: "encryptionkey@0"
I20260807 22:45:34.599015 19903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "e5a19ec0ee44430dae086a88fbd33245"
format_stamp: "Formatted at 2026-08-07 22:45:34 on dist-test-slave-rgb1"
server_key: "cdb7289a4dafc94df2797c9b04b14321"
server_key_iv: "ab6c517245639840a061412efc2b4701"
server_key_version: "encryptionkey@0"
I20260807 22:45:34.600746 19903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260807 22:45:34.601848 19919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:34.602136 19903 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:34.602244 19903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "e5a19ec0ee44430dae086a88fbd33245"
format_stamp: "Formatted at 2026-08-07 22:45:34 on dist-test-slave-rgb1"
server_key: "cdb7289a4dafc94df2797c9b04b14321"
server_key_iv: "ab6c517245639840a061412efc2b4701"
server_key_version: "encryptionkey@0"
I20260807 22:45:34.602313 19903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:34.617257 19903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:34.619892 19903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:34.620030 19903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:34.625005 19903 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:40183
I20260807 22:45:34.625108 19974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:40183 every 8 connection(s)
I20260807 22:45:34.625515 19903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:34.626727 19975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:34.629348 19975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245: Bootstrap starting.
I20260807 22:45:34.629962 19975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:34.630268 19975 log.cc:824] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:34.631107 19975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245: No bootstrap required, opened a new log
I20260807 22:45:34.632834 19975 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a19ec0ee44430dae086a88fbd33245" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40183 } }
I20260807 22:45:34.633001 19975 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:34.633070 19975 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a19ec0ee44430dae086a88fbd33245, State: Initialized, Role: FOLLOWER
I20260807 22:45:34.633241 19975 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a19ec0ee44430dae086a88fbd33245" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40183 } }
I20260807 22:45:34.633327 19975 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:34.633373 19975 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:34.633422 19975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:34.634269 19975 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a19ec0ee44430dae086a88fbd33245" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40183 } }
I20260807 22:45:34.634395 19975 leader_election.cc:302] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5a19ec0ee44430dae086a88fbd33245; no voters:
I20260807 22:45:34.634596 19975 leader_election.cc:288] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:34.634645 19978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:34.634873 19975 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:34.634899 19978 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [term 1 LEADER]: Becoming Leader. State: Replica: e5a19ec0ee44430dae086a88fbd33245, State: Running, Role: LEADER
I20260807 22:45:34.635120 19978 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a19ec0ee44430dae086a88fbd33245" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40183 } }
I20260807 22:45:34.635394 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 19903
I20260807 22:45:34.635519 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:34.635639 19980 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5a19ec0ee44430dae086a88fbd33245. Latest consensus state: current_term: 1 leader_uuid: "e5a19ec0ee44430dae086a88fbd33245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a19ec0ee44430dae086a88fbd33245" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40183 } } }
I20260807 22:45:34.635715 19980 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:34.635746 16778 external_mini_cluster.cc:1428] Setting key e79d02b06785e367d85356b12e9b690b
I20260807 22:45:34.635898 19979 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5a19ec0ee44430dae086a88fbd33245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5a19ec0ee44430dae086a88fbd33245" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40183 } } }
I20260807 22:45:34.635967 19979 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245 [sys.catalog]: This master's current role is: LEADER
W20260807 22:45:34.636999 19996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:45:34.637070 19996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:45:34.637164 19987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:34.637382 19987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:34.639042 19987 catalog_manager.cc:1347] Generated new cluster ID: 75400ad7d1474bc3bd2010c34e5ca072
I20260807 22:45:34.639096 19987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 22:45:34 dist-test-slave-rgb1 krb5kdc[19890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:34.649124 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:34.636268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35592 (local address 127.16.98.190:40183)
0807 22:45:34.636918 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:34.636928 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:34.636973 (+ 45us) server_negotiation.cc:407] Connection header received
0807 22:45:34.637100 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:34.637107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:34.637163 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:34.638465 (+ 1302us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:34.638562 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:34.639918 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:34.641082 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:34.641736 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:34.641855 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:34.644863 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:34.644867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:34.644867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:34.644880 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:34.646959 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:34.647414 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:34.647417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:34.647417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:34.647530 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:34.647748 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:34.647749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:34.647749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:34.648075 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:34.648225 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:34.648355 (+ 130us) server_negotiation.cc:299] Negotiation successful
0807 22:45:34.648864 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":60,"threads_started":1}
I20260807 22:45:34.651558 19987 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:34.652349 19987 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:34.661448 19987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5a19ec0ee44430dae086a88fbd33245: Generated new TSK 0
I20260807 22:45:34.661964 19987 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:45:34.701093 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:40183
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:34.793251 20001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:34.793485 20001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:34.793514 20001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:34.794700 20001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:34.794744 20001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:34.794770 20001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:45:34.796239 20001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:40183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:34.796458 20001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:34.796746 20001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:34.803848 20009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:34.803777 20007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:34.803892 20001 server_base.cc:1034] running on GCE node
W20260807 22:45:34.803825 20006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:34.804777 20001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:34.805100 20001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:34.806293 20001 hybrid_clock.cc:648] HybridClock initialized: now 1786142734806259 us; error 65 us; skew 500 ppm
Aug 07 22:45:34 dist-test-slave-rgb1 krb5kdc[19890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142734, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:34.809747 20001 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:45:34.810272 20001 webserver.cc:460] Webserver started at http://127.16.98.129:44659/ using document root <none> and password file <none>
I20260807 22:45:34.810559 20001 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:34.810631 20001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:34.810784 20001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:34.811896 20001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "a343f138e85c480b94f4dc928f3dd1ea"
format_stamp: "Formatted at 2026-08-07 22:45:34 on dist-test-slave-rgb1"
server_key: "86f0e29bc567f19a6171c9aef40cdab8"
server_key_iv: "4bf054f0da704f0a53444240cc5b6bf4"
server_key_version: "encryptionkey@0"
I20260807 22:45:34.812275 20001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "a343f138e85c480b94f4dc928f3dd1ea"
format_stamp: "Formatted at 2026-08-07 22:45:34 on dist-test-slave-rgb1"
server_key: "86f0e29bc567f19a6171c9aef40cdab8"
server_key_iv: "4bf054f0da704f0a53444240cc5b6bf4"
server_key_version: "encryptionkey@0"
I20260807 22:45:34.813872 20001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:45:34.814874 20016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:34.815281 20001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:34.815415 20001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "a343f138e85c480b94f4dc928f3dd1ea"
format_stamp: "Formatted at 2026-08-07 22:45:34 on dist-test-slave-rgb1"
server_key: "86f0e29bc567f19a6171c9aef40cdab8"
server_key_iv: "4bf054f0da704f0a53444240cc5b6bf4"
server_key_version: "encryptionkey@0"
I20260807 22:45:34.815501 20001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:34.839810 20001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:34.842811 20001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:34.843075 20001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:34.843369 20001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:34.843727 20001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:34.843781 20001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:34.843823 20001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:34.843853 20001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:34.851430 20001 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:35493
I20260807 22:45:34.851485 20135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:35493 every 8 connection(s)
I20260807 22:45:34.851850 20001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:45:34.858593 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 20001
I20260807 22:45:34.858698 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:45:34.858888 16778 external_mini_cluster.cc:1428] Setting key acdac8b1ef4ddbb04b5be384de26f092
Aug 07 22:45:34 dist-test-slave-rgb1 krb5kdc[19890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142734, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:34.863793 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:34.852752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:58729 (local address 127.16.98.190:40183)
0807 22:45:34.852803 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:34.852805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:34.853128 (+ 323us) server_negotiation.cc:407] Connection header received
0807 22:45:34.853611 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:34.853612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:34.853628 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:34.853691 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:34.854730 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:34.855273 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:34.856429 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:34.856764 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:34.856832 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:34.859646 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:34.859648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:34.859648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:34.859656 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:34.861861 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:34.862510 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:34.862511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:34.862511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:34.862574 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:34.862916 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:34.862917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:34.862917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:34.863064 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:34.863131 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:34.863557 (+ 426us) server_negotiation.cc:299] Negotiation successful
0807 22:45:34.863645 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 22:45:34.864061 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:34.852733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:40183 (local address 127.16.98.129:58729)
0807 22:45:34.852899 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:34.852938 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:34.853492 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:34.853748 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:34.853751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:34.854388 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:34.854615 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:34.854621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:34.855388 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:34.855389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:34.856302 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:34.856305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:34.856844 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:34.856846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:34.856966 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:34.857640 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:34.857648 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:34.859531 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:34.861987 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:34.862001 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:34.862004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:34.862403 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:34.862649 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:34.862650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:34.862651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:34.862828 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:34.863150 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:34.863159 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:34.863353 (+ 194us) client_negotiation.cc:769] Sending connection context
0807 22:45:34.863533 (+ 180us) client_negotiation.cc:240] Negotiation successful
0807 22:45:34.863762 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260807 22:45:34.864696 20136 heartbeater.cc:347] Connected to a master server at 127.16.98.190:40183
I20260807 22:45:34.864835 20136 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:34.865054 20136 heartbeater.cc:511] Master 127.16.98.190:40183 requested a full tablet report, sending...
I20260807 22:45:34.865753 19937 ts_manager.cc:194] Registered new tserver with Master: a343f138e85c480b94f4dc928f3dd1ea (127.16.98.129:35493)
I20260807 22:45:34.866762 19937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:58729
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:45:34.896665 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:40183
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:34.992856 20141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:34.993170 20141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:34.993225 20141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:34.994622 20141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:34.994710 20141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:34.994768 20141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:45:34.996541 20141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:40183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:34.996906 20141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:34.997210 20141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:35.004081 20147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:35.004168 20146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:35.004086 20149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:35.004586 20141 server_base.cc:1034] running on GCE node
I20260807 22:45:35.004848 20141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:35.005163 20141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:35.006480 20141 hybrid_clock.cc:648] HybridClock initialized: now 1786142735006469 us; error 39 us; skew 500 ppm
Aug 07 22:45:35 dist-test-slave-rgb1 krb5kdc[19890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142735, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:35.010248 20141 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:45:35.011073 20141 webserver.cc:460] Webserver started at http://127.16.98.130:45477/ using document root <none> and password file <none>
I20260807 22:45:35.011404 20141 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:35.011492 20141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:35.011823 20141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:35.013144 20141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "00c0043c6a7d40a2a60f607b6e9ff3c5"
format_stamp: "Formatted at 2026-08-07 22:45:35 on dist-test-slave-rgb1"
server_key: "1f9dc7593398a269a4ab069cefb4f399"
server_key_iv: "612ad8a2b1b6fbe9f67e52771cff835f"
server_key_version: "encryptionkey@0"
I20260807 22:45:35.013561 20141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "00c0043c6a7d40a2a60f607b6e9ff3c5"
format_stamp: "Formatted at 2026-08-07 22:45:35 on dist-test-slave-rgb1"
server_key: "1f9dc7593398a269a4ab069cefb4f399"
server_key_iv: "612ad8a2b1b6fbe9f67e52771cff835f"
server_key_version: "encryptionkey@0"
I20260807 22:45:35.015237 20141 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 22:45:35.016261 20156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:35.016471 20141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:35.016642 20141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "00c0043c6a7d40a2a60f607b6e9ff3c5"
format_stamp: "Formatted at 2026-08-07 22:45:35 on dist-test-slave-rgb1"
server_key: "1f9dc7593398a269a4ab069cefb4f399"
server_key_iv: "612ad8a2b1b6fbe9f67e52771cff835f"
server_key_version: "encryptionkey@0"
I20260807 22:45:35.016713 20141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:35.027410 20141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:35.029896 20141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:35.030021 20141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:35.030215 20141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:35.030562 20141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:35.030599 20141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:35.030622 20141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:35.030635 20141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:35.037520 20141 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:34945
I20260807 22:45:35.037611 20275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:34945 every 8 connection(s)
I20260807 22:45:35.037935 20141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:45:35.044032 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 20141
I20260807 22:45:35.044200 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:45:35.044451 16778 external_mini_cluster.cc:1428] Setting key 35b7ed7319b288438e812cb6c59ed9b3
Aug 07 22:45:35 dist-test-slave-rgb1 krb5kdc[19890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142735, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:35.051391 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:35.039124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:52609 (local address 127.16.98.190:40183)
0807 22:45:35.039220 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:35.039222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:35.039432 (+ 210us) server_negotiation.cc:407] Connection header received
0807 22:45:35.039942 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:35.039943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:35.039954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:35.040032 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:35.041171 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:35.041781 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:35.042989 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:35.043194 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:35.043264 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:35.047209 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:35.047210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:35.047211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:35.047218 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:35.049198 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:35.049844 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:35.049845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:35.049845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:35.049919 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:35.050338 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:35.050340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:35.050340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:35.050497 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:35.050570 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:35.051081 (+ 511us) server_negotiation.cc:299] Negotiation successful
0807 22:45:35.051292 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 22:45:35.051503 20278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:35.039058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:40183 (local address 127.16.98.130:52609)
0807 22:45:35.039253 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:35.039298 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:35.039823 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:35.040114 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:35.040117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:35.040772 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:35.041050 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:35.041056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:35.041973 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:35.041974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:35.042858 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:35.042861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:35.043367 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:35.043369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:35.043727 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:35.044721 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:35.044731 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:35.047067 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:35.049309 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:35.049318 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:35.049322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:35.049739 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:35.050000 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:35.050001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:35.050002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:35.050235 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:35.050585 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:35.050593 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:35.050794 (+ 201us) client_negotiation.cc:769] Sending connection context
0807 22:45:35.050941 (+ 147us) client_negotiation.cc:240] Negotiation successful
0807 22:45:35.051174 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
I20260807 22:45:35.052408 20276 heartbeater.cc:347] Connected to a master server at 127.16.98.190:40183
I20260807 22:45:35.052536 20276 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:35.052829 20276 heartbeater.cc:511] Master 127.16.98.190:40183 requested a full tablet report, sending...
I20260807 22:45:35.053475 19937 ts_manager.cc:194] Registered new tserver with Master: 00c0043c6a7d40a2a60f607b6e9ff3c5 (127.16.98.130:34945)
I20260807 22:45:35.053850 19937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:52609
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:45:35.085755 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:40183
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:35.179289 20281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:35.179587 20281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:35.179637 20281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:35.181138 20281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:35.181239 20281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:35.181296 20281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:45:35.183290 20281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:40183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:35.183634 20281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:35.183912 20281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:35.190222 20286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:35.190222 20287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:35.190480 20289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:35.190934 20281 server_base.cc:1034] running on GCE node
I20260807 22:45:35.191177 20281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:35.191447 20281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:35.192667 20281 hybrid_clock.cc:648] HybridClock initialized: now 1786142735192630 us; error 64 us; skew 500 ppm
Aug 07 22:45:35 dist-test-slave-rgb1 krb5kdc[19890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142735, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:35.196012 20281 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:45:35.196550 20281 webserver.cc:460] Webserver started at http://127.16.98.131:39449/ using document root <none> and password file <none>
I20260807 22:45:35.196883 20281 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:35.196954 20281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:35.197120 20281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:35.198297 20281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "6cb60f9347ab476e95ddcebfce5c4205"
format_stamp: "Formatted at 2026-08-07 22:45:35 on dist-test-slave-rgb1"
server_key: "cd20019725b5962cc207d834fbf0e403"
server_key_iv: "cd1d0374a38e1dab60d12c24b03781d0"
server_key_version: "encryptionkey@0"
I20260807 22:45:35.198684 20281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "6cb60f9347ab476e95ddcebfce5c4205"
format_stamp: "Formatted at 2026-08-07 22:45:35 on dist-test-slave-rgb1"
server_key: "cd20019725b5962cc207d834fbf0e403"
server_key_iv: "cd1d0374a38e1dab60d12c24b03781d0"
server_key_version: "encryptionkey@0"
I20260807 22:45:35.201071 20281 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 22:45:35.202214 20296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:35.202621 20281 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:35.202877 20281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "6cb60f9347ab476e95ddcebfce5c4205"
format_stamp: "Formatted at 2026-08-07 22:45:35 on dist-test-slave-rgb1"
server_key: "cd20019725b5962cc207d834fbf0e403"
server_key_iv: "cd1d0374a38e1dab60d12c24b03781d0"
server_key_version: "encryptionkey@0"
I20260807 22:45:35.203058 20281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:35.218182 20281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:35.221449 20281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:35.221598 20281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:35.221875 20281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:35.222349 20281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:35.222393 20281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:35.222419 20281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:35.222467 20281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:35.231534 20281 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:46459
I20260807 22:45:35.231627 20415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:46459 every 8 connection(s)
I20260807 22:45:35.232024 20281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:45:35.232648 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 20281
I20260807 22:45:35.232767 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:45:35.232983 16778 external_mini_cluster.cc:1428] Setting key e70a2bbd0f9fbc06e82df21ed1dace29
Aug 07 22:45:35 dist-test-slave-rgb1 krb5kdc[19890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142735, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:35.243543 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:35.233095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:34985 (local address 127.16.98.190:40183)
0807 22:45:35.233158 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:35.233161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:35.233417 (+ 256us) server_negotiation.cc:407] Connection header received
0807 22:45:35.234096 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:35.234097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:35.234110 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:35.234155 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:35.235248 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:35.235836 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:35.237053 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:35.237287 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:35.237384 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:35.239781 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:35.239783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:35.239783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:35.239798 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:35.241429 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:35.241858 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:35.241859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:35.241859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:35.241980 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:35.242573 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:35.242574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:35.242574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:35.242710 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:35.242787 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:35.243301 (+ 514us) server_negotiation.cc:299] Negotiation successful
0807 22:45:35.243450 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 22:45:35.243642 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:35.233119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:40183 (local address 127.16.98.131:34985)
0807 22:45:35.233298 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:35.233344 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:35.233996 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:35.234228 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:35.234232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:35.234829 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:35.235148 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:35.235153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:35.235941 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:35.235942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:35.236943 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:35.236946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:35.237399 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:35.237400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:35.237541 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:35.238118 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:35.238127 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:35.239655 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:35.241521 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:35.241528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:35.241530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:35.241775 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:35.242217 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:35.242219 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:35.242220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:35.242443 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:35.242910 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:35.242919 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:35.243066 (+ 147us) client_negotiation.cc:769] Sending connection context
0807 22:45:35.243186 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 22:45:35.243426 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
I20260807 22:45:35.244562 20417 heartbeater.cc:347] Connected to a master server at 127.16.98.190:40183
I20260807 22:45:35.244745 20417 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:35.244992 20417 heartbeater.cc:511] Master 127.16.98.190:40183 requested a full tablet report, sending...
I20260807 22:45:35.245492 19937 ts_manager.cc:194] Registered new tserver with Master: 6cb60f9347ab476e95ddcebfce5c4205 (127.16.98.131:46459)
I20260807 22:45:35.245977 19937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:34985
I20260807 22:45:35.257123 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 22:45:35 dist-test-slave-rgb1 krb5kdc[19890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142735, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 07 22:45:35 dist-test-slave-rgb1 krb5kdc[19890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142735, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:35.286556 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:35.276710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.16.98.190:40183)
0807 22:45:35.276788 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:35.276790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:35.276822 (+ 32us) server_negotiation.cc:407] Connection header received
0807 22:45:35.276927 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:35.276928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:35.276941 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:35.277000 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:35.277749 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:35.278287 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:35.279150 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:35.279335 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:35.279395 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:35.282856 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:35.282858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:35.282858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:35.282869 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:35.284824 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:35.285280 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:35.285281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:35.285282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:35.285340 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:35.285582 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:35.285583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:35.285583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:35.285763 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:35.285846 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:35.286135 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 22:45:35.286464 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260807 22:45:35.286966 19937 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35602
I20260807 22:45:35.287665 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 20001
I20260807 22:45:35.294466 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 20141
I20260807 22:45:35.300124 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 20281
I20260807 22:45:35.305994 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 19903
2026-08-07T22:45:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2134 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:45:35 dist-test-slave-rgb1 krb5kdc[20429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:45:35 dist-test-slave-rgb1 krb5kdc[20429](info): set up 2 sockets
Aug 07 22:45:35 dist-test-slave-rgb1 krb5kdc[20429](info): commencing operation
krb5kdc: starting...
W20260807 22:45:36.474830 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:45:36 dist-test-slave-rgb1 krb5kdc[20429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:45:36.624143 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:35919
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:35919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:36.717327 20443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:36.717628 20443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:36.717679 20443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:36.719107 20443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:36.719189 20443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:36.719216 20443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:36.719244 20443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:36.719275 20443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:36.721488 20443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:35919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:35919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:36.721882 20443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:36.722196 20443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:36.728144 20448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:36.728142 20451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:36.728341 20449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:36.728387 20443 server_base.cc:1034] running on GCE node
I20260807 22:45:36.728806 20443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:36.729175 20443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:36.730377 20443 hybrid_clock.cc:648] HybridClock initialized: now 1786142736730353 us; error 44 us; skew 500 ppm
Aug 07 22:45:36 dist-test-slave-rgb1 krb5kdc[20429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142736, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:36.734076 20443 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:45:36.734704 20443 webserver.cc:460] Webserver started at http://127.16.98.190:45467/ using document root <none> and password file <none>
I20260807 22:45:36.734999 20443 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:36.735080 20443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:36.735257 20443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:36.736512 20443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "1db9fc10cad04402b25c54ba5841f43f"
format_stamp: "Formatted at 2026-08-07 22:45:36 on dist-test-slave-rgb1"
server_key: "c28565e74fe2260a60f2ab53c8a1df1b"
server_key_iv: "b4eab40488e35e01842812deaeb0bdaa"
server_key_version: "encryptionkey@0"
I20260807 22:45:36.736994 20443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "1db9fc10cad04402b25c54ba5841f43f"
format_stamp: "Formatted at 2026-08-07 22:45:36 on dist-test-slave-rgb1"
server_key: "c28565e74fe2260a60f2ab53c8a1df1b"
server_key_iv: "b4eab40488e35e01842812deaeb0bdaa"
server_key_version: "encryptionkey@0"
I20260807 22:45:36.738582 20443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:45:36.739734 20458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:36.740132 20443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:36.740268 20443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "1db9fc10cad04402b25c54ba5841f43f"
format_stamp: "Formatted at 2026-08-07 22:45:36 on dist-test-slave-rgb1"
server_key: "c28565e74fe2260a60f2ab53c8a1df1b"
server_key_iv: "b4eab40488e35e01842812deaeb0bdaa"
server_key_version: "encryptionkey@0"
I20260807 22:45:36.740360 20443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:36.753309 20443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:36.756319 20443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:36.756652 20443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:36.762009 20443 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:35919
I20260807 22:45:36.762045 20511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:35919 every 8 connection(s)
I20260807 22:45:36.762673 20443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:36.763423 20512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:36.765944 20512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f: Bootstrap starting.
I20260807 22:45:36.766604 20512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:36.766950 20512 log.cc:824] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:36.767750 20512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f: No bootstrap required, opened a new log
I20260807 22:45:36.769556 20512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db9fc10cad04402b25c54ba5841f43f" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35919 } }
I20260807 22:45:36.769702 20512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:36.769726 20512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db9fc10cad04402b25c54ba5841f43f, State: Initialized, Role: FOLLOWER
I20260807 22:45:36.769897 20512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db9fc10cad04402b25c54ba5841f43f" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35919 } }
I20260807 22:45:36.769981 20512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:36.770007 20512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:36.770056 20512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:36.771076 20512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db9fc10cad04402b25c54ba5841f43f" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35919 } }
I20260807 22:45:36.771175 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 20443
I20260807 22:45:36.771198 20512 leader_election.cc:302] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1db9fc10cad04402b25c54ba5841f43f; no voters:
I20260807 22:45:36.771278 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:36.771399 20512 leader_election.cc:288] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:36.771488 16778 external_mini_cluster.cc:1428] Setting key e8af4fcd65c80c204ad88179e28bf531
I20260807 22:45:36.771493 20515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:36.771695 20515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [term 1 LEADER]: Becoming Leader. State: Replica: 1db9fc10cad04402b25c54ba5841f43f, State: Running, Role: LEADER
I20260807 22:45:36.771816 20515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db9fc10cad04402b25c54ba5841f43f" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35919 } }
I20260807 22:45:36.771699 20512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:36.772346 20517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1db9fc10cad04402b25c54ba5841f43f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db9fc10cad04402b25c54ba5841f43f" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35919 } } }
I20260807 22:45:36.772526 20517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:36.773517 20521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:36.773693 20516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1db9fc10cad04402b25c54ba5841f43f. Latest consensus state: current_term: 1 leader_uuid: "1db9fc10cad04402b25c54ba5841f43f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db9fc10cad04402b25c54ba5841f43f" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35919 } } }
I20260807 22:45:36.773787 20516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:36.774355 20521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:36.776471 20521 catalog_manager.cc:1347] Generated new cluster ID: ea777af4c4c24277a53e32cecd05f845
I20260807 22:45:36.776537 20521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 22:45:36 dist-test-slave-rgb1 krb5kdc[20429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:36.786096 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:36.772052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.16.98.190:35919)
0807 22:45:36.772251 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:36.772263 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:36.772306 (+ 43us) server_negotiation.cc:407] Connection header received
0807 22:45:36.772468 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:36.772478 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:36.772587 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:36.773382 (+ 795us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:36.774332 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:36.775857 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:36.777074 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:36.777790 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:36.777925 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:36.781205 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:36.781210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:36.781211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:36.781226 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:36.783907 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:36.784450 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:36.784453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:36.784454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:36.784620 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:36.784869 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:36.784870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:36.784870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:36.785242 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:36.785420 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:36.785558 (+ 138us) server_negotiation.cc:299] Negotiation successful
0807 22:45:36.785718 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260807 22:45:36.807097 20521 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:36.807876 20521 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:36.826445 20521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1db9fc10cad04402b25c54ba5841f43f: Generated new TSK 0
I20260807 22:45:36.826726 20521 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:45:36.860991 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:35919
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:36.956766 20541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:36.956995 20541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:36.957026 20541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:36.958257 20541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:36.958310 20541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:36.958339 20541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:45:36.959935 20541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:35919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:36.960167 20541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:36.960366 20541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:36.966598 20546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:36.966575 20549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:36.966859 20541 server_base.cc:1034] running on GCE node
W20260807 22:45:36.966674 20547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:36.967232 20541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:36.967526 20541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:36.968672 20541 hybrid_clock.cc:648] HybridClock initialized: now 1786142736968644 us; error 86 us; skew 500 ppm
Aug 07 22:45:36 dist-test-slave-rgb1 krb5kdc[20429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142736, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:36.972029 20541 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:45:36.972666 20541 webserver.cc:460] Webserver started at http://127.16.98.129:46079/ using document root <none> and password file <none>
I20260807 22:45:36.972911 20541 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:36.972991 20541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:36.973168 20541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:36.974460 20541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "67fd0788956a4eae97d5d1e98ba9eb23"
format_stamp: "Formatted at 2026-08-07 22:45:36 on dist-test-slave-rgb1"
server_key: "0148f2a851841b50b143f6d5bf9787fa"
server_key_iv: "2b127b451ab7dbfb087b8ee6b1ee865f"
server_key_version: "encryptionkey@0"
I20260807 22:45:36.974864 20541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "67fd0788956a4eae97d5d1e98ba9eb23"
format_stamp: "Formatted at 2026-08-07 22:45:36 on dist-test-slave-rgb1"
server_key: "0148f2a851841b50b143f6d5bf9787fa"
server_key_iv: "2b127b451ab7dbfb087b8ee6b1ee865f"
server_key_version: "encryptionkey@0"
I20260807 22:45:36.976435 20541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:45:36.977439 20556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:36.977626 20541 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:36.977720 20541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "67fd0788956a4eae97d5d1e98ba9eb23"
format_stamp: "Formatted at 2026-08-07 22:45:36 on dist-test-slave-rgb1"
server_key: "0148f2a851841b50b143f6d5bf9787fa"
server_key_iv: "2b127b451ab7dbfb087b8ee6b1ee865f"
server_key_version: "encryptionkey@0"
I20260807 22:45:36.977778 20541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:36.993968 20541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:36.996416 20541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:36.996662 20541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:36.996932 20541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:36.997315 20541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:36.997372 20541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:36.997426 20541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:36.997454 20541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:37.004599 20541 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:46481
I20260807 22:45:37.004642 20670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:46481 every 8 connection(s)
I20260807 22:45:37.005023 20541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:45:37.007655 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 20541
I20260807 22:45:37.007787 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:45:37.008044 16778 external_mini_cluster.cc:1428] Setting key 2b62d8827bae317a9b69dcff95bdadd0
Aug 07 22:45:37 dist-test-slave-rgb1 krb5kdc[20429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142736, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:37.018427 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:37.005980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:39613 (local address 127.16.98.190:35919)
0807 22:45:37.006057 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:37.006059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:37.006453 (+ 394us) server_negotiation.cc:407] Connection header received
0807 22:45:37.007440 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:37.007442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:37.007461 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:37.007550 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:37.008465 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.009206 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.011083 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.011515 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.011611 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:37.014360 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:37.014361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:37.014362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:37.014373 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:37.016539 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:37.017141 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:37.017143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:37.017143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:37.017217 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:37.017558 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:37.017559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:37.017559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:37.017731 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:37.017813 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:37.018171 (+ 358us) server_negotiation.cc:299] Negotiation successful
0807 22:45:37.018283 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 22:45:37.018555 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:37.005884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35919 (local address 127.16.98.129:39613)
0807 22:45:37.006169 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:37.006220 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:37.007267 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:37.007635 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:37.007640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:37.008037 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:37.008334 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:37.008341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.009328 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.009329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:37.010910 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:37.010914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.011636 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.011637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:37.011778 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:37.012542 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:37.012598 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:37.014238 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:37.016702 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:37.016713 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:37.016715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:37.017044 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:37.017308 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:37.017309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:37.017309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:37.017471 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:37.017829 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:37.017837 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:37.017979 (+ 142us) client_negotiation.cc:769] Sending connection context
0807 22:45:37.018116 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 22:45:37.018283 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
I20260807 22:45:37.019162 20671 heartbeater.cc:347] Connected to a master server at 127.16.98.190:35919
I20260807 22:45:37.019289 20671 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:37.019502 20671 heartbeater.cc:511] Master 127.16.98.190:35919 requested a full tablet report, sending...
I20260807 22:45:37.020401 20475 ts_manager.cc:194] Registered new tserver with Master: 67fd0788956a4eae97d5d1e98ba9eb23 (127.16.98.129:46481)
I20260807 22:45:37.021225 20475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:39613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:45:37.041145 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:35919
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:37.133627 20677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:37.133876 20677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:37.133908 20677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:37.135299 20677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:37.135394 20677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:37.135447 20677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:45:37.137207 20677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:35919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:37.137531 20677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:37.137820 20677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:37.143888 20685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:37.144063 20682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:37.144029 20683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:37.144724 20677 server_base.cc:1034] running on GCE node
I20260807 22:45:37.144958 20677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:37.145244 20677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:37.146420 20677 hybrid_clock.cc:648] HybridClock initialized: now 1786142737146380 us; error 50 us; skew 500 ppm
Aug 07 22:45:37 dist-test-slave-rgb1 krb5kdc[20429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142737, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:37.149961 20677 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:45:37.150782 20677 webserver.cc:460] Webserver started at http://127.16.98.130:43037/ using document root <none> and password file <none>
I20260807 22:45:37.151139 20677 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:37.151222 20677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:37.151398 20677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:37.152803 20677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "80e89d4fba14439ca281afbd765da7c5"
format_stamp: "Formatted at 2026-08-07 22:45:37 on dist-test-slave-rgb1"
server_key: "106aa5deaf4efaa8e38ba90410d1c6ee"
server_key_iv: "274ff9d577e2571ad6e795c4d35036dc"
server_key_version: "encryptionkey@0"
I20260807 22:45:37.153261 20677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "80e89d4fba14439ca281afbd765da7c5"
format_stamp: "Formatted at 2026-08-07 22:45:37 on dist-test-slave-rgb1"
server_key: "106aa5deaf4efaa8e38ba90410d1c6ee"
server_key_iv: "274ff9d577e2571ad6e795c4d35036dc"
server_key_version: "encryptionkey@0"
I20260807 22:45:37.155228 20677 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 22:45:37.156505 20692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:37.156857 20677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:37.156997 20677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "80e89d4fba14439ca281afbd765da7c5"
format_stamp: "Formatted at 2026-08-07 22:45:37 on dist-test-slave-rgb1"
server_key: "106aa5deaf4efaa8e38ba90410d1c6ee"
server_key_iv: "274ff9d577e2571ad6e795c4d35036dc"
server_key_version: "encryptionkey@0"
I20260807 22:45:37.157087 20677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:37.169178 20677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:37.172775 20677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:37.173048 20677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:37.173461 20677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:37.173935 20677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:37.174103 20677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:37.174177 20677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:37.174213 20677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:37.181489 20677 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:38351
I20260807 22:45:37.181586 20814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:38351 every 8 connection(s)
I20260807 22:45:37.182026 20677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:45:37.188735 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 20677
I20260807 22:45:37.188891 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:45:37.189126 16778 external_mini_cluster.cc:1428] Setting key 3a408ff48564d082c9a1832e3afbecc4
Aug 07 22:45:37 dist-test-slave-rgb1 krb5kdc[20429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142737, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:37.194339 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:37.182914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:57643 (local address 127.16.98.190:35919)
0807 22:45:37.183003 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:37.183006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:37.183240 (+ 234us) server_negotiation.cc:407] Connection header received
0807 22:45:37.184312 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:37.184313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:37.184325 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:37.184390 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:37.185261 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.185857 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.187048 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.187267 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.187385 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:37.190407 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:37.190408 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:37.190409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:37.190418 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:37.192420 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:37.192987 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:37.192989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:37.192990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:37.193069 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:37.193506 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:37.193508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:37.193509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:37.193653 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:37.193750 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:37.194108 (+ 358us) server_negotiation.cc:299] Negotiation successful
0807 22:45:37.194236 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 22:45:37.194566 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:37.182901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35919 (local address 127.16.98.130:57643)
0807 22:45:37.183103 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:37.183145 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:37.184161 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:37.184464 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:37.184468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:37.184853 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:37.185156 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:37.185162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.185969 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.185970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:37.186944 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:37.186947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.187369 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.187370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:37.187476 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:37.188170 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:37.188179 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:37.190287 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:37.192526 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:37.192536 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:37.192544 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:37.192893 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:37.193167 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:37.193167 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:37.193168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:37.193305 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:37.193741 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:37.193750 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:37.193916 (+ 166us) client_negotiation.cc:769] Sending connection context
0807 22:45:37.194066 (+ 150us) client_negotiation.cc:240] Negotiation successful
0807 22:45:37.194269 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
I20260807 22:45:37.195190 20815 heartbeater.cc:347] Connected to a master server at 127.16.98.190:35919
I20260807 22:45:37.195297 20815 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:37.195537 20815 heartbeater.cc:511] Master 127.16.98.190:35919 requested a full tablet report, sending...
I20260807 22:45:37.195988 20475 ts_manager.cc:194] Registered new tserver with Master: 80e89d4fba14439ca281afbd765da7c5 (127.16.98.130:38351)
I20260807 22:45:37.196444 20475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:57643
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:45:37.225281 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:35919
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:37.318840 20821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:37.319067 20821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:37.319093 20821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:37.320276 20821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:37.320320 20821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:37.320348 20821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:45:37.322023 20821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:35919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:37.322243 20821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:37.322453 20821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:37.327950 20828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:37.327970 20830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:37.327972 20827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:37.328392 20821 server_base.cc:1034] running on GCE node
I20260807 22:45:37.328688 20821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:37.328948 20821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:37.330142 20821 hybrid_clock.cc:648] HybridClock initialized: now 1786142737330112 us; error 47 us; skew 500 ppm
Aug 07 22:45:37 dist-test-slave-rgb1 krb5kdc[20429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142737, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:37.333312 20821 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:45:37.333887 20821 webserver.cc:460] Webserver started at http://127.16.98.131:45473/ using document root <none> and password file <none>
I20260807 22:45:37.334376 20821 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:37.334440 20821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:37.334584 20821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:37.335891 20821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "27cd438a4c414554b357b30038c3aebc"
format_stamp: "Formatted at 2026-08-07 22:45:37 on dist-test-slave-rgb1"
server_key: "5110b2efccbd3c737e5a6bc623fbeb18"
server_key_iv: "8f474ecb82169b67514b36651321cb45"
server_key_version: "encryptionkey@0"
I20260807 22:45:37.336261 20821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "27cd438a4c414554b357b30038c3aebc"
format_stamp: "Formatted at 2026-08-07 22:45:37 on dist-test-slave-rgb1"
server_key: "5110b2efccbd3c737e5a6bc623fbeb18"
server_key_iv: "8f474ecb82169b67514b36651321cb45"
server_key_version: "encryptionkey@0"
I20260807 22:45:37.337946 20821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260807 22:45:37.338917 20837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:37.339202 20821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:37.339337 20821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "27cd438a4c414554b357b30038c3aebc"
format_stamp: "Formatted at 2026-08-07 22:45:37 on dist-test-slave-rgb1"
server_key: "5110b2efccbd3c737e5a6bc623fbeb18"
server_key_iv: "8f474ecb82169b67514b36651321cb45"
server_key_version: "encryptionkey@0"
I20260807 22:45:37.339435 20821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:37.352706 20821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:37.355146 20821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:37.355383 20821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:37.355701 20821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:37.356092 20821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:37.356156 20821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:37.356204 20821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:37.356259 20821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:37.363740 20821 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:37491
I20260807 22:45:37.363796 20954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:37491 every 8 connection(s)
I20260807 22:45:37.364501 20821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:45:37.372763 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 20821
I20260807 22:45:37.372876 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
Aug 07 22:45:37 dist-test-slave-rgb1 krb5kdc[20429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142737, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:37.373100 16778 external_mini_cluster.cc:1428] Setting key 7b3a98c5e6971659547041ec09d1c132
I20260807 22:45:37.377420 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:37.365670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:47891 (local address 127.16.98.190:35919)
0807 22:45:37.365772 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:37.365774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:37.366132 (+ 358us) server_negotiation.cc:407] Connection header received
0807 22:45:37.367128 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:37.367130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:37.367160 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:37.367230 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:37.368020 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.368635 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.370137 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.370373 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.370457 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:37.373610 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:37.373613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:37.373613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:37.373622 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:37.375767 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:37.376276 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:37.376277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:37.376277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:37.376347 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:37.376686 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:37.376687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:37.376687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:37.376825 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:37.376908 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:37.377227 (+ 319us) server_negotiation.cc:299] Negotiation successful
0807 22:45:37.377321 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 22:45:37.377667 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:37.365626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35919 (local address 127.16.98.131:47891)
0807 22:45:37.365826 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:37.365875 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:37.366971 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:37.367322 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:37.367326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:37.367670 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:37.367885 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:37.367890 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.368742 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.368743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:37.369956 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:37.369964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.370524 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.370526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:37.370686 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:37.371486 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:37.371496 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:37.373500 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:37.375890 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:37.375906 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:37.375916 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:37.376184 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:37.376420 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:37.376421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:37.376421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:37.376599 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:37.376898 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:37.376906 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:37.377064 (+ 158us) client_negotiation.cc:769] Sending connection context
0807 22:45:37.377204 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 22:45:37.377388 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
I20260807 22:45:37.378232 20955 heartbeater.cc:347] Connected to a master server at 127.16.98.190:35919
I20260807 22:45:37.378351 20955 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:37.378530 20955 heartbeater.cc:511] Master 127.16.98.190:35919 requested a full tablet report, sending...
I20260807 22:45:37.378978 20475 ts_manager.cc:194] Registered new tserver with Master: 27cd438a4c414554b357b30038c3aebc (127.16.98.131:37491)
I20260807 22:45:37.379384 20475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:47891
I20260807 22:45:37.386276 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 22:45:37 dist-test-slave-rgb1 krb5kdc[20429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 22:45:37 dist-test-slave-rgb1 krb5kdc[20429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.129@KRBTEST.COM
I20260807 22:45:37.417984 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:37.407103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.16.98.129:46481)
0807 22:45:37.407366 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:37.407372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:37.407384 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:45:37.407415 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:37.407417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:37.407434 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:37.407520 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:37.408288 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.409286 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.410146 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.410514 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.410605 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:37.413525 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:37.413549 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:37.413549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:37.413563 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:37.416161 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:37.416657 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:37.416661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:37.416661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:37.416744 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:37.417067 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:37.417069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:37.417069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:37.417377 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:37.417523 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:37.417808 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 22:45:37.417894 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":140,"threads_started":1}
Aug 07 22:45:37 dist-test-slave-rgb1 krb5kdc[20429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:37.428771 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:37.419960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58196 (local address 127.16.98.190:35919)
0807 22:45:37.420072 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:37.420074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:37.420116 (+ 42us) server_negotiation.cc:407] Connection header received
0807 22:45:37.420145 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:37.420146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:37.420157 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:37.420208 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:37.421208 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.421751 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.422658 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:37.422873 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:37.422941 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:37.425801 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:37.425803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:37.425803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:37.425810 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:37.427241 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:37.427627 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:37.427628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:37.427628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:37.427690 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:37.428091 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:37.428092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:37.428092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:37.428268 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:37.428359 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:37.428547 (+ 188us) server_negotiation.cc:299] Negotiation successful
0807 22:45:37.428698 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260807 22:45:37.429124 20475 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58196
I20260807 22:45:37.429546 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 20541
I20260807 22:45:37.437510 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 20677
I20260807 22:45:37.443415 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 20821
I20260807 22:45:37.448784 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 20443
2026-08-07T22:45:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2141 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:45:37 dist-test-slave-rgb1 krb5kdc[20968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:45:37 dist-test-slave-rgb1 krb5kdc[20968](info): set up 2 sockets
Aug 07 22:45:37 dist-test-slave-rgb1 krb5kdc[20968](info): commencing operation
krb5kdc: starting...
W20260807 22:45:38.566062 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:45:38 dist-test-slave-rgb1 krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:45:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:45:38.714092 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:34211
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:34417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:34211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:38.806784 20981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:38.807030 20981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:38.807057 20981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:38.808338 20981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:38.808391 20981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:38.808406 20981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:38.808419 20981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:38.808432 20981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:38.810166 20981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:34211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:34211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:38.810434 20981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:38.810724 20981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:38.816805 20986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:38.817044 20987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:38.817044 20990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:38.817387 20981 server_base.cc:1034] running on GCE node
I20260807 22:45:38.817667 20981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:38.818002 20981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:38.819190 20981 hybrid_clock.cc:648] HybridClock initialized: now 1786142738819158 us; error 42 us; skew 500 ppm
Aug 07 22:45:38 dist-test-slave-rgb1 krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142738, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:38.828457 20981 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:45:38.828958 20981 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:38.829022 20981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:38.829137 20981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:38.830425 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "783e3e6f181b4987b8a7bc5c0c94b430"
format_stamp: "Formatted at 2026-08-07 22:45:38 on dist-test-slave-rgb1"
server_key: "de891ad021d90a05e68ce8f3607a66b4"
server_key_iv: "2af108464fd08f1aff6c7e18d4a8cdaa"
server_key_version: "encryptionkey@0"
I20260807 22:45:38.830892 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "783e3e6f181b4987b8a7bc5c0c94b430"
format_stamp: "Formatted at 2026-08-07 22:45:38 on dist-test-slave-rgb1"
server_key: "de891ad021d90a05e68ce8f3607a66b4"
server_key_iv: "2af108464fd08f1aff6c7e18d4a8cdaa"
server_key_version: "encryptionkey@0"
I20260807 22:45:38.832485 20981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:45:38.833793 20996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:38.834045 20981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:38.834141 20981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "783e3e6f181b4987b8a7bc5c0c94b430"
format_stamp: "Formatted at 2026-08-07 22:45:38 on dist-test-slave-rgb1"
server_key: "de891ad021d90a05e68ce8f3607a66b4"
server_key_iv: "2af108464fd08f1aff6c7e18d4a8cdaa"
server_key_version: "encryptionkey@0"
I20260807 22:45:38.834194 20981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:38.852252 20981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:38.856383 20981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:38.856598 20981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:38.861806 20981 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:34211
I20260807 22:45:38.861837 21051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:34211 every 8 connection(s)
I20260807 22:45:38.862221 20981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:38.863866 21052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:38.867200 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430: Bootstrap starting.
I20260807 22:45:38.868105 21052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:38.868494 21052 log.cc:824] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:38.870044 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430: No bootstrap required, opened a new log
I20260807 22:45:38.871757 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 20981
I20260807 22:45:38.871939 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:38.872159 16778 external_mini_cluster.cc:1428] Setting key f4a330fa0bf3202fcca6c2d94a504c9e
I20260807 22:45:38.873202 21052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783e3e6f181b4987b8a7bc5c0c94b430" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34211 } }
I20260807 22:45:38.873474 21052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:38.873525 21052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783e3e6f181b4987b8a7bc5c0c94b430, State: Initialized, Role: FOLLOWER
I20260807 22:45:38.873718 21052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783e3e6f181b4987b8a7bc5c0c94b430" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34211 } }
I20260807 22:45:38.873821 21052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:38.873867 21052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:38.873924 21052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:38.875104 21052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783e3e6f181b4987b8a7bc5c0c94b430" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34211 } }
I20260807 22:45:38.875464 21052 leader_election.cc:302] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 783e3e6f181b4987b8a7bc5c0c94b430; no voters:
I20260807 22:45:38.875779 21052 leader_election.cc:288] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:38.876003 21057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:38.876209 21057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [term 1 LEADER]: Becoming Leader. State: Replica: 783e3e6f181b4987b8a7bc5c0c94b430, State: Running, Role: LEADER
I20260807 22:45:38.876324 21052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:38.876350 21057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783e3e6f181b4987b8a7bc5c0c94b430" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34211 } }
I20260807 22:45:38.876870 21059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "783e3e6f181b4987b8a7bc5c0c94b430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783e3e6f181b4987b8a7bc5c0c94b430" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34211 } } }
I20260807 22:45:38.877053 21059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:38.877888 21058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 783e3e6f181b4987b8a7bc5c0c94b430. Latest consensus state: current_term: 1 leader_uuid: "783e3e6f181b4987b8a7bc5c0c94b430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783e3e6f181b4987b8a7bc5c0c94b430" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34211 } } }
I20260807 22:45:38.877986 21067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:38.878165 21058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:38.878985 21067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:38.880990 21067 catalog_manager.cc:1347] Generated new cluster ID: a97ff0f03a314412aad0c3bcef51f858
I20260807 22:45:38.881050 21067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 22:45:38 dist-test-slave-rgb1 krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:38.887650 21067 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:38.888377 21067 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:38.889246 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:38.873480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.16.98.190:34211)
0807 22:45:38.873799 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:38.873808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:38.873954 (+ 146us) server_negotiation.cc:407] Connection header received
0807 22:45:38.874672 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:38.874682 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:38.874752 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:38.874902 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:38.876361 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:38.878974 (+ 2613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:38.880212 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:38.880869 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:38.881330 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:38.884111 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:38.884116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:38.884117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:38.884132 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:38.887092 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:38.887607 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:38.887610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:38.887611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:38.887744 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:38.888057 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:38.888059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:38.888059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:38.888377 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:38.888551 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:38.888818 (+ 267us) server_negotiation.cc:299] Negotiation successful
0807 22:45:38.889007 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":205,"threads_started":1}
I20260807 22:45:38.895934 21067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430: Generated new TSK 0
I20260807 22:45:38.896181 21067 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:45:38.947573 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34211
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:34417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:39.059015 21079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:39.059252 21079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:39.059281 21079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:39.060637 21079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:39.060688 21079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:39.060719 21079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:45:39.062508 21079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:39.062762 21079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:39.063050 21079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:39.069401 21087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:39.069437 21084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:39.069445 21085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:39.069787 21079 server_base.cc:1034] running on GCE node
I20260807 22:45:39.070080 21079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:39.070441 21079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:39.071661 21079 hybrid_clock.cc:648] HybridClock initialized: now 1786142739071662 us; error 50 us; skew 500 ppm
Aug 07 22:45:39 dist-test-slave-rgb1 krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:39.076102 21079 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:45:39.076668 21079 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:39.076777 21079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:39.076944 21079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:39.078328 21079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "3f37fa8d33de4abbb4e1371f7f44e523"
format_stamp: "Formatted at 2026-08-07 22:45:39 on dist-test-slave-rgb1"
server_key: "930c6265c9d96d9d2563515a1454c541"
server_key_iv: "0bebfc16a3acacca5cd07966ebe66b04"
server_key_version: "encryptionkey@0"
I20260807 22:45:39.078728 21079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "3f37fa8d33de4abbb4e1371f7f44e523"
format_stamp: "Formatted at 2026-08-07 22:45:39 on dist-test-slave-rgb1"
server_key: "930c6265c9d96d9d2563515a1454c541"
server_key_iv: "0bebfc16a3acacca5cd07966ebe66b04"
server_key_version: "encryptionkey@0"
I20260807 22:45:39.080250 21079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:45:39.081792 21093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:39.082302 21079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:39.082430 21079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "3f37fa8d33de4abbb4e1371f7f44e523"
format_stamp: "Formatted at 2026-08-07 22:45:39 on dist-test-slave-rgb1"
server_key: "930c6265c9d96d9d2563515a1454c541"
server_key_iv: "0bebfc16a3acacca5cd07966ebe66b04"
server_key_version: "encryptionkey@0"
I20260807 22:45:39.082544 21079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:39.113247 21079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:39.116204 21079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:39.116405 21079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:39.116760 21079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:39.117118 21079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:39.117175 21079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:39.117218 21079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:39.117249 21079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:39.125200 21079 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:39333
I20260807 22:45:39.125239 21212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:39333 every 8 connection(s)
I20260807 22:45:39.125567 21079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:45:39.127314 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 21079
I20260807 22:45:39.127511 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:45:39.127727 16778 external_mini_cluster.cc:1428] Setting key b926484fe3f347b70f497b703e7eef6b
Aug 07 22:45:39 dist-test-slave-rgb1 krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
I20260807 22:45:39.139556 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.126456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:43425 (local address 127.16.98.190:34211)
0807 22:45:39.126538 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:39.126540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:39.127010 (+ 470us) server_negotiation.cc:407] Connection header received
0807 22:45:39.127841 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:39.127843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:39.127859 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:39.127952 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:39.129273 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.129876 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.131816 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.132091 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.132197 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.134826 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:39.134828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:39.134828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:39.134837 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:39.136764 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:39.137419 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:39.137422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:39.137422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:39.137621 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:39.138424 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:39.138425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:39.138426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:39.138753 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:39.138868 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:39.139269 (+ 401us) server_negotiation.cc:299] Negotiation successful
0807 22:45:39.139385 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 22:45:39.139847 21215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.126436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:34211 (local address 127.16.98.129:43425)
0807 22:45:39.126836 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:39.126882 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:39.127679 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:39.128028 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:39.128033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:39.128450 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:39.129155 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.129162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.130159 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.130161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.131676 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.131679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.132216 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.132217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.132360 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.133021 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:39.133030 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:39.134692 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:39.137001 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:39.137011 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:39.137014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:39.137288 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:39.138141 (+ 853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:39.138142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:39.138142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:39.138348 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:39.138859 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:39.138868 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:39.139089 (+ 221us) client_negotiation.cc:769] Sending connection context
0807 22:45:39.139249 (+ 160us) client_negotiation.cc:240] Negotiation successful
0807 22:45:39.139455 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":101,"threads_started":1}
I20260807 22:45:39.140782 21213 heartbeater.cc:347] Connected to a master server at 127.16.98.190:34211
I20260807 22:45:39.140924 21213 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:39.141158 21213 heartbeater.cc:511] Master 127.16.98.190:34211 requested a full tablet report, sending...
I20260807 22:45:39.142390 21013 ts_manager.cc:194] Registered new tserver with Master: 3f37fa8d33de4abbb4e1371f7f44e523 (127.16.98.129:39333)
I20260807 22:45:39.143210 21013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:43425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:45:39.175561 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34211
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:34417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:39.274788 21218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:39.275053 21218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:39.275084 21218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:39.276489 21218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:39.276543 21218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:39.276645 21218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:45:39.278671 21218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:39.278975 21218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:39.279253 21218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:39.286005 21223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:39.286031 21226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:39.286149 21218 server_base.cc:1034] running on GCE node
W20260807 22:45:39.286088 21224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:39.286562 21218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:39.286955 21218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:39.288195 21218 hybrid_clock.cc:648] HybridClock initialized: now 1786142739288181 us; error 38 us; skew 500 ppm
Aug 07 22:45:39 dist-test-slave-rgb1 krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:39.292019 21218 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:45:39.292516 21218 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:39.292624 21218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:39.292811 21218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:39.294034 21218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "9cfbf9bc0cec4f05822014d02368c6cb"
format_stamp: "Formatted at 2026-08-07 22:45:39 on dist-test-slave-rgb1"
server_key: "cd1b9a871a9d862d6652364db00bab5b"
server_key_iv: "8ee3c47a8839727b5c6300666bfa2c55"
server_key_version: "encryptionkey@0"
I20260807 22:45:39.294423 21218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "9cfbf9bc0cec4f05822014d02368c6cb"
format_stamp: "Formatted at 2026-08-07 22:45:39 on dist-test-slave-rgb1"
server_key: "cd1b9a871a9d862d6652364db00bab5b"
server_key_iv: "8ee3c47a8839727b5c6300666bfa2c55"
server_key_version: "encryptionkey@0"
I20260807 22:45:39.296376 21218 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260807 22:45:39.297782 21232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:39.298182 21218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:39.298324 21218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "9cfbf9bc0cec4f05822014d02368c6cb"
format_stamp: "Formatted at 2026-08-07 22:45:39 on dist-test-slave-rgb1"
server_key: "cd1b9a871a9d862d6652364db00bab5b"
server_key_iv: "8ee3c47a8839727b5c6300666bfa2c55"
server_key_version: "encryptionkey@0"
I20260807 22:45:39.298441 21218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:39.318969 21218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:39.321768 21218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:39.321993 21218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:39.322319 21218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:39.322698 21218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:39.322760 21218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:39.322810 21218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:39.322844 21218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:39.330468 21218 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:44809
I20260807 22:45:39.330531 21347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:44809 every 8 connection(s)
I20260807 22:45:39.330885 21218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:45:39.334281 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 21218
I20260807 22:45:39.334399 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:45:39.334622 16778 external_mini_cluster.cc:1428] Setting key e731b0ad30b7ac074c781c679a218171
Aug 07 22:45:39 dist-test-slave-rgb1 krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:39.342028 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.331827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:32917 (local address 127.16.98.190:34211)
0807 22:45:39.331913 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:39.331915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:39.332147 (+ 232us) server_negotiation.cc:407] Connection header received
0807 22:45:39.332811 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:39.332814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:39.332834 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:39.332952 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:39.334248 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.335007 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.336219 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.336460 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.336552 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.338992 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:39.338994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:39.338994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:39.339002 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:39.340483 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:39.340947 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:39.340948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:39.340955 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:39.341017 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:39.341285 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:39.341286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:39.341286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:39.341412 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:39.341497 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:39.341823 (+ 326us) server_negotiation.cc:299] Negotiation successful
0807 22:45:39.341923 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 22:45:39.342270 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.331782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:34211 (local address 127.16.98.130:32917)
0807 22:45:39.331967 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:39.332025 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:39.332545 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:39.333115 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:39.333119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:39.333437 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:39.334094 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.334101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.335122 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.335124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.336116 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.336119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.336601 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.336603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.336734 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.337366 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:39.337375 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:39.338878 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:39.340594 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:39.340604 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:39.340607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:39.340851 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:39.341087 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:39.341087 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:39.341088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:39.341210 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:39.341486 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:39.341494 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:39.341683 (+ 189us) client_negotiation.cc:769] Sending connection context
0807 22:45:39.341791 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 22:45:39.341965 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260807 22:45:39.343103 21348 heartbeater.cc:347] Connected to a master server at 127.16.98.190:34211
I20260807 22:45:39.343377 21348 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:39.343633 21348 heartbeater.cc:511] Master 127.16.98.190:34211 requested a full tablet report, sending...
I20260807 22:45:39.344200 21013 ts_manager.cc:194] Registered new tserver with Master: 9cfbf9bc0cec4f05822014d02368c6cb (127.16.98.130:44809)
I20260807 22:45:39.344985 21013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:32917
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:45:39.378144 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34211
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:34417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:39.477859 21353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:39.478132 21353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:39.478179 21353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:39.479408 21353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:39.479452 21353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:39.479482 21353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:45:39.481105 21353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:39.481309 21353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:39.481498 21353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:39.486917 21361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:39.487071 21358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:39.486917 21359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:39.487529 21353 server_base.cc:1034] running on GCE node
I20260807 22:45:39.487756 21353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:39.488000 21353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:39.489262 21353 hybrid_clock.cc:648] HybridClock initialized: now 1786142739489210 us; error 66 us; skew 500 ppm
Aug 07 22:45:39 dist-test-slave-rgb1 krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:39.492805 21353 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:45:39.493204 21353 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:39.493258 21353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:39.493399 21353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:39.494516 21353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "5f3625f945c640efaac0181bebee0840"
format_stamp: "Formatted at 2026-08-07 22:45:39 on dist-test-slave-rgb1"
server_key: "739b81578946d51409bbc5ba4ac40064"
server_key_iv: "056977d8875ee7a1efad2f1611bcd508"
server_key_version: "encryptionkey@0"
I20260807 22:45:39.494889 21353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "5f3625f945c640efaac0181bebee0840"
format_stamp: "Formatted at 2026-08-07 22:45:39 on dist-test-slave-rgb1"
server_key: "739b81578946d51409bbc5ba4ac40064"
server_key_iv: "056977d8875ee7a1efad2f1611bcd508"
server_key_version: "encryptionkey@0"
I20260807 22:45:39.496277 21353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:45:39.497330 21367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:39.497646 21353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:39.497753 21353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "5f3625f945c640efaac0181bebee0840"
format_stamp: "Formatted at 2026-08-07 22:45:39 on dist-test-slave-rgb1"
server_key: "739b81578946d51409bbc5ba4ac40064"
server_key_iv: "056977d8875ee7a1efad2f1611bcd508"
server_key_version: "encryptionkey@0"
I20260807 22:45:39.497843 21353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:39.506063 21353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:39.508734 21353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:39.508948 21353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:39.509251 21353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:39.509653 21353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:39.509714 21353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:39.509754 21353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:39.509802 21353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:39.516840 21353 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:44319
I20260807 22:45:39.516924 21485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:44319 every 8 connection(s)
I20260807 22:45:39.517262 21353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
Aug 07 22:45:39 dist-test-slave-rgb1 krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:39.525695 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 21353
I20260807 22:45:39.525820 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:45:39.526047 16778 external_mini_cluster.cc:1428] Setting key 59b1ab7da36cff3e2391ef9060ee2a4e
I20260807 22:45:39.528679 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.518393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:40433 (local address 127.16.98.190:34211)
0807 22:45:39.518475 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:39.518477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:39.518508 (+ 31us) server_negotiation.cc:407] Connection header received
0807 22:45:39.519109 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:39.519110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:39.519123 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:39.519188 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:39.520267 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.520863 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.522084 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.522276 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.522342 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.525104 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:39.525105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:39.525105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:39.525113 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:39.526922 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:39.527420 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:39.527421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:39.527422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:39.527493 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:39.527824 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:39.527825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:39.527825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:39.528031 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:39.528102 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:39.528443 (+ 341us) server_negotiation.cc:299] Negotiation successful
0807 22:45:39.528524 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 22:45:39.528863 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.518054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:34211 (local address 127.16.98.131:40433)
0807 22:45:39.518400 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:39.518440 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:39.518935 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:39.519298 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:39.519302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:39.519603 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:39.520141 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.520147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.521049 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.521050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.521948 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.521951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.522381 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.522382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.522522 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.523249 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:39.523258 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:39.524984 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:39.527005 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:39.527016 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:39.527017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:39.527349 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:39.527594 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:39.527595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:39.527595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:39.527733 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:39.528141 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:39.528148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:39.528280 (+ 132us) client_negotiation.cc:769] Sending connection context
0807 22:45:39.528389 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 22:45:39.528539 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":83,"threads_started":1}
I20260807 22:45:39.529580 21486 heartbeater.cc:347] Connected to a master server at 127.16.98.190:34211
I20260807 22:45:39.529809 21486 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:39.529976 21486 heartbeater.cc:511] Master 127.16.98.190:34211 requested a full tablet report, sending...
I20260807 22:45:39.530637 21013 ts_manager.cc:194] Registered new tserver with Master: 5f3625f945c640efaac0181bebee0840 (127.16.98.131:44319)
I20260807 22:45:39.531016 21013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:40433
I20260807 22:45:39.538844 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:45:39.547577 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.540798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.16.98.190:34211)
0807 22:45:39.540877 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:39.540879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:39.540991 (+ 112us) server_negotiation.cc:407] Connection header received
0807 22:45:39.541118 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:39.541119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:39.541130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:39.541208 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:39.541853 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.542441 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.543180 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.543383 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.543470 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.544830 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:39.544832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:39.544832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:39.544839 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:39.546286 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:39.546682 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:39.546683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:39.546684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:39.546756 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:39.546970 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:39.546971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:39.546971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:39.547157 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:39.547267 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:39.547383 (+ 116us) server_negotiation.cc:299] Negotiation successful
0807 22:45:39.547470 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 22:45:39.550545 21013 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:45:39.552695 21013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:39.561038 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.556594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:44809 (local address 127.0.0.1:49352)
0807 22:45:39.556822 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:39.556867 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:39.557146 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:39.557450 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:39.557453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:39.557456 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:39.557534 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.557537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.558916 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.558917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.559942 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.559944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.560777 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.560778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.560847 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.560859 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 22:45:39.560923 (+ 64us) client_negotiation.cc:240] Negotiation successful
0807 22:45:39.560970 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260807 22:45:39.561395 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.556726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49352 (local address 127.16.98.130:44809)
0807 22:45:39.556932 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:39.556938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:39.557263 (+ 325us) server_negotiation.cc:407] Connection header received
0807 22:45:39.557317 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:39.557320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:39.557337 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:39.557419 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:39.557629 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.558766 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.560037 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.560657 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.560772 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.560857 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:39.560960 (+ 103us) server_negotiation.cc:299] Negotiation successful
0807 22:45:39.561009 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
I20260807 22:45:39.561911 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.556667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:44319 (local address 127.0.0.1:46414)
0807 22:45:39.557004 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:39.557017 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:39.557092 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:39.557712 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:39.557713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:39.557716 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:39.557771 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.557773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.558707 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.558708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.560602 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.560605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.561634 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.561635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.561706 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.561711 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:45:39.561787 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 22:45:39.561839 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":93,"thread_start_us":99,"threads_started":1}
I20260807 22:45:39.562256 21501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.556925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.16.98.131:44319)
0807 22:45:39.557275 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:39.557281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:39.557292 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:45:39.557331 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:39.557333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:39.557617 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:39.557698 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:39.557834 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.558589 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.560688 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.561491 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.561612 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.561674 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:39.562125 (+ 451us) server_negotiation.cc:299] Negotiation successful
0807 22:45:39.562176 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":94,"threads_started":1}
I20260807 22:45:39.562254 21280 tablet_service.cc:1467] Processing CreateTablet for tablet d74af130a8c24005ac61e20aba3e5a58 (DEFAULT_TABLE table=test-table [id=56594d1468fc458c88efb7acccb3cb9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:39.562680 21280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74af130a8c24005ac61e20aba3e5a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:39.562780 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.556667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:39333 (local address 127.0.0.1:46170)
0807 22:45:39.556994 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:39.557006 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:39.557169 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:39.557511 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:39.557511 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:39.557513 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:39.557552 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.557553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.558888 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.558888 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.560622 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.560623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.562529 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.562530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.562596 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.562600 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:39.562659 (+ 59us) client_negotiation.cc:240] Negotiation successful
0807 22:45:39.562710 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":4,"thread_start_us":103,"threads_started":1}
I20260807 22:45:39.562793 21502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.556990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.16.98.129:39333)
0807 22:45:39.557204 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:39.557208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:39.557219 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:45:39.557263 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:39.557266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:39.557282 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:39.557418 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:39.557629 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.558792 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.561718 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.562441 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.562527 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.562580 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:39.562673 (+ 93us) server_negotiation.cc:299] Negotiation successful
0807 22:45:39.562712 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":92,"threads_started":1}
I20260807 22:45:39.563411 21142 tablet_service.cc:1467] Processing CreateTablet for tablet d74af130a8c24005ac61e20aba3e5a58 (DEFAULT_TABLE table=test-table [id=56594d1468fc458c88efb7acccb3cb9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:39.563717 21142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74af130a8c24005ac61e20aba3e5a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:39.565894 21503 tablet_bootstrap.cc:492] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb: Bootstrap starting.
I20260807 22:45:39.566581 21503 tablet_bootstrap.cc:654] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:39.566625 21417 tablet_service.cc:1467] Processing CreateTablet for tablet d74af130a8c24005ac61e20aba3e5a58 (DEFAULT_TABLE table=test-table [id=56594d1468fc458c88efb7acccb3cb9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:39.566980 21417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74af130a8c24005ac61e20aba3e5a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:39.566977 21504 tablet_bootstrap.cc:492] T d74af130a8c24005ac61e20aba3e5a58 P 3f37fa8d33de4abbb4e1371f7f44e523: Bootstrap starting.
I20260807 22:45:39.567261 21503 log.cc:824] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:39.567669 21504 tablet_bootstrap.cc:654] T d74af130a8c24005ac61e20aba3e5a58 P 3f37fa8d33de4abbb4e1371f7f44e523: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:39.568111 21503 tablet_bootstrap.cc:492] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb: No bootstrap required, opened a new log
I20260807 22:45:39.568200 21503 ts_tablet_manager.cc:1397] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:45:39.568282 21504 log.cc:824] T d74af130a8c24005ac61e20aba3e5a58 P 3f37fa8d33de4abbb4e1371f7f44e523: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:39.569298 21504 tablet_bootstrap.cc:492] T d74af130a8c24005ac61e20aba3e5a58 P 3f37fa8d33de4abbb4e1371f7f44e523: No bootstrap required, opened a new log
I20260807 22:45:39.569402 21504 ts_tablet_manager.cc:1397] T d74af130a8c24005ac61e20aba3e5a58 P 3f37fa8d33de4abbb4e1371f7f44e523: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:45:39.570570 21503 raft_consensus.cc:348] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfbf9bc0cec4f05822014d02368c6cb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44809 } } peers { permanent_uuid: "5f3625f945c640efaac0181bebee0840" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44319 } } peers { permanent_uuid: "3f37fa8d33de4abbb4e1371f7f44e523" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39333 } }
I20260807 22:45:39.570813 21503 raft_consensus.cc:374] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:39.570931 21503 raft_consensus.cc:729] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cfbf9bc0cec4f05822014d02368c6cb, State: Initialized, Role: FOLLOWER
I20260807 22:45:39.571043 21507 tablet_bootstrap.cc:492] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840: Bootstrap starting.
I20260807 22:45:39.571098 21503 consensus_queue.cc:260] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfbf9bc0cec4f05822014d02368c6cb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44809 } } peers { permanent_uuid: "5f3625f945c640efaac0181bebee0840" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44319 } } peers { permanent_uuid: "3f37fa8d33de4abbb4e1371f7f44e523" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39333 } }
I20260807 22:45:39.571467 21503 ts_tablet_manager.cc:1428] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:45:39.571637 21348 heartbeater.cc:503] Master 127.16.98.190:34211 was elected leader, sending a full tablet report...
I20260807 22:45:39.571978 21507 tablet_bootstrap.cc:654] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:39.571880 21504 raft_consensus.cc:348] T d74af130a8c24005ac61e20aba3e5a58 P 3f37fa8d33de4abbb4e1371f7f44e523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfbf9bc0cec4f05822014d02368c6cb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44809 } } peers { permanent_uuid: "5f3625f945c640efaac0181bebee0840" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44319 } } peers { permanent_uuid: "3f37fa8d33de4abbb4e1371f7f44e523" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39333 } }
I20260807 22:45:39.572055 21504 raft_consensus.cc:374] T d74af130a8c24005ac61e20aba3e5a58 P 3f37fa8d33de4abbb4e1371f7f44e523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:39.572108 21504 raft_consensus.cc:729] T d74af130a8c24005ac61e20aba3e5a58 P 3f37fa8d33de4abbb4e1371f7f44e523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f37fa8d33de4abbb4e1371f7f44e523, State: Initialized, Role: FOLLOWER
I20260807 22:45:39.572252 21504 consensus_queue.cc:260] T d74af130a8c24005ac61e20aba3e5a58 P 3f37fa8d33de4abbb4e1371f7f44e523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfbf9bc0cec4f05822014d02368c6cb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44809 } } peers { permanent_uuid: "5f3625f945c640efaac0181bebee0840" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44319 } } peers { permanent_uuid: "3f37fa8d33de4abbb4e1371f7f44e523" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39333 } }
I20260807 22:45:39.572700 21504 ts_tablet_manager.cc:1428] T d74af130a8c24005ac61e20aba3e5a58 P 3f37fa8d33de4abbb4e1371f7f44e523: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260807 22:45:39.572769 21213 heartbeater.cc:503] Master 127.16.98.190:34211 was elected leader, sending a full tablet report...
I20260807 22:45:39.573104 21507 log.cc:824] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:39.574213 21507 tablet_bootstrap.cc:492] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840: No bootstrap required, opened a new log
I20260807 22:45:39.574322 21507 ts_tablet_manager.cc:1397] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 22:45:39.576869 21507 raft_consensus.cc:348] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfbf9bc0cec4f05822014d02368c6cb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44809 } } peers { permanent_uuid: "5f3625f945c640efaac0181bebee0840" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44319 } } peers { permanent_uuid: "3f37fa8d33de4abbb4e1371f7f44e523" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39333 } }
I20260807 22:45:39.577077 21507 raft_consensus.cc:374] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:39.577122 21507 raft_consensus.cc:729] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3625f945c640efaac0181bebee0840, State: Initialized, Role: FOLLOWER
I20260807 22:45:39.577243 21507 consensus_queue.cc:260] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfbf9bc0cec4f05822014d02368c6cb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44809 } } peers { permanent_uuid: "5f3625f945c640efaac0181bebee0840" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44319 } } peers { permanent_uuid: "3f37fa8d33de4abbb4e1371f7f44e523" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39333 } }
I20260807 22:45:39.577603 21507 ts_tablet_manager.cc:1428] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260807 22:45:39.577684 21486 heartbeater.cc:503] Master 127.16.98.190:34211 was elected leader, sending a full tablet report...
W20260807 22:45:39.581611 21349 tablet.cc:2369] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:45:39.626576 21214 tablet.cc:2369] T d74af130a8c24005ac61e20aba3e5a58 P 3f37fa8d33de4abbb4e1371f7f44e523: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:45:39.634013 21511 raft_consensus.cc:482] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:39.634176 21511 raft_consensus.cc:504] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfbf9bc0cec4f05822014d02368c6cb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44809 } } peers { permanent_uuid: "5f3625f945c640efaac0181bebee0840" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44319 } } peers { permanent_uuid: "3f37fa8d33de4abbb4e1371f7f44e523" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39333 } }
I20260807 22:45:39.634649 21511 leader_election.cc:288] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cfbf9bc0cec4f05822014d02368c6cb (127.16.98.130:44809), 3f37fa8d33de4abbb4e1371f7f44e523 (127.16.98.129:39333)
I20260807 22:45:39.638993 21502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.635105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:43725 (local address 127.16.98.129:39333)
0807 22:45:39.635185 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:39.635188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:39.635285 (+ 97us) server_negotiation.cc:407] Connection header received
0807 22:45:39.635359 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:39.635360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:39.635370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:39.635455 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:39.635660 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.636265 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.638114 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.638527 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.638600 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.638609 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:39.638863 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 22:45:39.638916 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 22:45:39.639113 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.635006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:39333 (local address 127.16.98.131:43725)
0807 22:45:39.635198 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:39.635210 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:39.635297 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:39.635457 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:39.635458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:39.635460 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:39.635545 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.635547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.636364 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.636366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.637986 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.637990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.638657 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.638659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.638733 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.638737 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:39.638838 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 22:45:39.638890 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":60,"thread_start_us":111,"threads_started":1}
I20260807 22:45:39.639493 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.634857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:44809 (local address 127.16.98.131:52827)
0807 22:45:39.634920 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:39.634934 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:39.634997 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:39.635181 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:39.635183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:39.635185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:39.635270 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.635272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.636416 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.636417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.637986 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:39.637990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.639199 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.639200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:39.639313 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.639317 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:39.639368 (+ 51us) client_negotiation.cc:240] Negotiation successful
0807 22:45:39.639409 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":107}
I20260807 22:45:39.639578 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.634961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:52827 (local address 127.16.98.130:44809)
0807 22:45:39.635020 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:39.635022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:39.635052 (+ 30us) server_negotiation.cc:407] Connection header received
0807 22:45:39.635087 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:39.635088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:39.635099 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:39.635153 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:39.635384 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.636308 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.638644 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.639029 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.639139 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.639156 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:39.639380 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 22:45:39.639437 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260807 22:45:39.639627 21162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74af130a8c24005ac61e20aba3e5a58" candidate_uuid: "5f3625f945c640efaac0181bebee0840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f37fa8d33de4abbb4e1371f7f44e523" is_pre_election: true
I20260807 22:45:39.639873 21162 raft_consensus.cc:2454] T d74af130a8c24005ac61e20aba3e5a58 P 3f37fa8d33de4abbb4e1371f7f44e523 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f3625f945c640efaac0181bebee0840 in term 0.
I20260807 22:45:39.639899 21300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74af130a8c24005ac61e20aba3e5a58" candidate_uuid: "5f3625f945c640efaac0181bebee0840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cfbf9bc0cec4f05822014d02368c6cb" is_pre_election: true
I20260807 22:45:39.640058 21300 raft_consensus.cc:2454] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f3625f945c640efaac0181bebee0840 in term 0.
I20260807 22:45:39.640252 21371 leader_election.cc:302] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f37fa8d33de4abbb4e1371f7f44e523, 5f3625f945c640efaac0181bebee0840; no voters:
I20260807 22:45:39.640444 21511 raft_consensus.cc:2781] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:39.640518 21511 raft_consensus.cc:482] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:39.640549 21511 raft_consensus.cc:3037] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:39.641376 21511 raft_consensus.cc:504] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfbf9bc0cec4f05822014d02368c6cb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44809 } } peers { permanent_uuid: "5f3625f945c640efaac0181bebee0840" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44319 } } peers { permanent_uuid: "3f37fa8d33de4abbb4e1371f7f44e523" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39333 } }
I20260807 22:45:39.641532 21511 leader_election.cc:288] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [CANDIDATE]: Term 1 election: Requested vote from peers 9cfbf9bc0cec4f05822014d02368c6cb (127.16.98.130:44809), 3f37fa8d33de4abbb4e1371f7f44e523 (127.16.98.129:39333)
I20260807 22:45:39.641705 21162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74af130a8c24005ac61e20aba3e5a58" candidate_uuid: "5f3625f945c640efaac0181bebee0840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f37fa8d33de4abbb4e1371f7f44e523"
I20260807 22:45:39.641788 21162 raft_consensus.cc:3037] T d74af130a8c24005ac61e20aba3e5a58 P 3f37fa8d33de4abbb4e1371f7f44e523 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:39.641784 21300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74af130a8c24005ac61e20aba3e5a58" candidate_uuid: "5f3625f945c640efaac0181bebee0840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cfbf9bc0cec4f05822014d02368c6cb"
I20260807 22:45:39.642014 21300 raft_consensus.cc:3037] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:39.642961 21300 raft_consensus.cc:2454] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f3625f945c640efaac0181bebee0840 in term 1.
I20260807 22:45:39.642990 21162 raft_consensus.cc:2454] T d74af130a8c24005ac61e20aba3e5a58 P 3f37fa8d33de4abbb4e1371f7f44e523 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f3625f945c640efaac0181bebee0840 in term 1.
I20260807 22:45:39.643221 21370 leader_election.cc:302] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f3625f945c640efaac0181bebee0840, 9cfbf9bc0cec4f05822014d02368c6cb; no voters:
I20260807 22:45:39.643356 21511 raft_consensus.cc:2781] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:39.643585 21511 raft_consensus.cc:686] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [term 1 LEADER]: Becoming Leader. State: Replica: 5f3625f945c640efaac0181bebee0840, State: Running, Role: LEADER
I20260807 22:45:39.643718 21511 consensus_queue.cc:237] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfbf9bc0cec4f05822014d02368c6cb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44809 } } peers { permanent_uuid: "5f3625f945c640efaac0181bebee0840" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44319 } } peers { permanent_uuid: "3f37fa8d33de4abbb4e1371f7f44e523" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39333 } }
I20260807 22:45:39.644714 21012 catalog_manager.cc:5495] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f3625f945c640efaac0181bebee0840 (127.16.98.131). New cstate: current_term: 1 leader_uuid: "5f3625f945c640efaac0181bebee0840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfbf9bc0cec4f05822014d02368c6cb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f3625f945c640efaac0181bebee0840" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f37fa8d33de4abbb4e1371f7f44e523" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39333 } health_report { overall_health: UNKNOWN } } }
I20260807 22:45:39.662290 21501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.659038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46420 (local address 127.16.98.131:44319)
0807 22:45:39.659115 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:39.659117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:39.659126 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:45:39.659179 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:39.659180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:39.659201 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:39.659290 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:39.659493 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.660078 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.661209 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:39.661450 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:39.661547 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:39.661786 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:39.661952 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:39.662077 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:39.662107 (+ 30us) server_negotiation.cc:299] Negotiation successful
0807 22:45:39.662135 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 22:45:39.665066 21300 raft_consensus.cc:1261] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb [term 1 FOLLOWER]: Refusing update from remote peer 5f3625f945c640efaac0181bebee0840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:39.665120 21162 raft_consensus.cc:1261] T d74af130a8c24005ac61e20aba3e5a58 P 3f37fa8d33de4abbb4e1371f7f44e523 [term 1 FOLLOWER]: Refusing update from remote peer 5f3625f945c640efaac0181bebee0840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:39.665841 21513 consensus_queue.cc:1035] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f37fa8d33de4abbb4e1371f7f44e523" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:39.666002 21511 consensus_queue.cc:1035] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cfbf9bc0cec4f05822014d02368c6cb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:39.669775 21516 mvcc.cc:205] Tried to move back new op lower bound from 7316040661662879744 to 7316040661581438976. Current Snapshot: MvccSnapshot[applied={T|T < 7316040661662879744}]
I20260807 22:45:39.671253 21517 mvcc.cc:205] Tried to move back new op lower bound from 7316040661662879744 to 7316040661581438976. Current Snapshot: MvccSnapshot[applied={T|T < 7316040661662879744}]
I20260807 22:45:39.671438 21518 mvcc.cc:205] Tried to move back new op lower bound from 7316040661662879744 to 7316040661581438976. Current Snapshot: MvccSnapshot[applied={T|T < 7316040661662879744 or (T in {7316040661662879744})}]
I20260807 22:45:39.674535 21013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:39.674656 21013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:39.675675 21013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 783e3e6f181b4987b8a7bc5c0c94b430: Sending DeleteTablet for 3 replicas of tablet d74af130a8c24005ac61e20aba3e5a58
I20260807 22:45:39.676084 21417 tablet_service.cc:1514] Processing DeleteTablet for tablet d74af130a8c24005ac61e20aba3e5a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:39 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:46414
I20260807 22:45:39.676096 21280 tablet_service.cc:1514] Processing DeleteTablet for tablet d74af130a8c24005ac61e20aba3e5a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:39 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:49352
I20260807 22:45:39.676177 21142 tablet_service.cc:1514] Processing DeleteTablet for tablet d74af130a8c24005ac61e20aba3e5a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:39 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:46170
I20260807 22:45:39.676411 21527 tablet_replica.cc:330] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840: stopping tablet replica
I20260807 22:45:39.676528 21527 raft_consensus.cc:2229] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:45:39.676633 21526 tablet_replica.cc:330] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb: stopping tablet replica
I20260807 22:45:39.676721 21527 raft_consensus.cc:2258] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:39.676734 21526 raft_consensus.cc:2229] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:39.677011 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 21079
I20260807 22:45:39.677244 21527 ts_tablet_manager.cc:1905] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:39.679636 21527 ts_tablet_manager.cc:1918] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:39.679728 21527 log.cc:1196] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/d74af130a8c24005ac61e20aba3e5a58
I20260807 22:45:39.680043 21527 ts_tablet_manager.cc:1939] T d74af130a8c24005ac61e20aba3e5a58 P 5f3625f945c640efaac0181bebee0840: Deleting consensus metadata
I20260807 22:45:39.680806 21000 catalog_manager.cc:4841] TS 5f3625f945c640efaac0181bebee0840 (127.16.98.131:44319): tablet d74af130a8c24005ac61e20aba3e5a58 (table test-table [id=56594d1468fc458c88efb7acccb3cb9a]) successfully deleted
I20260807 22:45:39.684796 21526 raft_consensus.cc:2258] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:39.685428 21526 ts_tablet_manager.cc:1905] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 22:45:39.686667 20999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:39333 (error 108)
I20260807 22:45:39.687335 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 21218
I20260807 22:45:39.687845 21526 ts_tablet_manager.cc:1918] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:39.687974 21526 log.cc:1196] T d74af130a8c24005ac61e20aba3e5a58 P 9cfbf9bc0cec4f05822014d02368c6cb: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/d74af130a8c24005ac61e20aba3e5a58
I20260807 22:45:39.688714 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.687597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:39333 (local address 127.0.0.1:46176)
0807 22:45:39.688527 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:39.688660 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:39333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":892}
W20260807 22:45:39.689136 20999 catalog_manager.cc:4584] TS 3f37fa8d33de4abbb4e1371f7f44e523 (127.16.98.129:39333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d74af130a8c24005ac61e20aba3e5a58: Network error: Client connection negotiation failed: client connection to 127.16.98.129:39333: connect: Connection refused (error 111)
I20260807 22:45:39.696372 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 21353
I20260807 22:45:39.698863 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.696411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:44809 (local address 127.0.0.1:49366)
0807 22:45:39.698776 (+ 2365us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:39.698824 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.130:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2338}
I20260807 22:45:39.702155 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 20981
2026-08-07T22:45:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2254 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-07T22:45:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:39Z Disabled control of system clock
I20260807 22:45:39.735212 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:34155
--webserver_interface=127.16.98.190
--webserver_port=0
--builtin_ntp_servers=127.16.98.148:34825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:34155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 22:45:39.827889 21533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:39.828140 21533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:39.828163 21533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:39.829447 21533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:39.829501 21533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:39.829517 21533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:39.829531 21533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:39.829543 21533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:39.831115 21533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:34155
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:34155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:39.831419 21533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:39.831775 21533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:39.837275 21541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:39.837263 21539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:39.837262 21538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:39.837486 21533 server_base.cc:1034] running on GCE node
I20260807 22:45:39.837921 21533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:39.838308 21533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:39.839496 21533 hybrid_clock.cc:648] HybridClock initialized: now 1786142739839439 us; error 72 us; skew 500 ppm
I20260807 22:45:39.841023 21533 webserver.cc:460] Webserver started at http://127.16.98.190:39789/ using document root <none> and password file <none>
I20260807 22:45:39.841295 21533 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:39.841370 21533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:39.841580 21533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:39.842764 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "2256d458b2734c8d8075801a03a46ced"
format_stamp: "Formatted at 2026-08-07 22:45:39 on dist-test-slave-rgb1"
server_key: "9b0474a37c77b81d4f59cf39af7ce896"
server_key_iv: "e130eb65c38bb4d791f2ce6986ae7d21"
server_key_version: "encryptionkey@0"
I20260807 22:45:39.843227 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "2256d458b2734c8d8075801a03a46ced"
format_stamp: "Formatted at 2026-08-07 22:45:39 on dist-test-slave-rgb1"
server_key: "9b0474a37c77b81d4f59cf39af7ce896"
server_key_iv: "e130eb65c38bb4d791f2ce6986ae7d21"
server_key_version: "encryptionkey@0"
I20260807 22:45:39.844947 21533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260807 22:45:39.846377 21547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:39.846742 21533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:39.846905 21533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "2256d458b2734c8d8075801a03a46ced"
format_stamp: "Formatted at 2026-08-07 22:45:39 on dist-test-slave-rgb1"
server_key: "9b0474a37c77b81d4f59cf39af7ce896"
server_key_iv: "e130eb65c38bb4d791f2ce6986ae7d21"
server_key_version: "encryptionkey@0"
I20260807 22:45:39.847082 21533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:39.851579 21533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:39.851946 21533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:39.852129 21533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:39.857182 21533 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:34155
I20260807 22:45:39.857291 21601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:34155 every 8 connection(s)
I20260807 22:45:39.857697 21533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:39.858906 21602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:39.861208 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 21533
I20260807 22:45:39.861351 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:39.861563 16778 external_mini_cluster.cc:1428] Setting key b12e5e89565d92376573e5138556c2bc
I20260807 22:45:39.863051 21602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced: Bootstrap starting.
I20260807 22:45:39.863934 21602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:39.864315 21602 log.cc:824] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:39.864996 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:39.862255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.16.98.190:34155)
0807 22:45:39.862515 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:39.862526 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:39.862568 (+ 42us) server_negotiation.cc:407] Connection header received
0807 22:45:39.863190 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:39.863200 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:39.863259 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:39.863462 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:39.864414 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:39.864418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:39.864419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:39.864424 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:39.864479 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:39.864494 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:39.864625 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:39.864699 (+ 74us) server_negotiation.cc:299] Negotiation successful
0807 22:45:39.864763 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":98,"threads_started":1}
I20260807 22:45:39.865285 21602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced: No bootstrap required, opened a new log
I20260807 22:45:39.866909 21602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2256d458b2734c8d8075801a03a46ced" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34155 } }
I20260807 22:45:39.867072 21602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:39.867096 21602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2256d458b2734c8d8075801a03a46ced, State: Initialized, Role: FOLLOWER
I20260807 22:45:39.867223 21602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2256d458b2734c8d8075801a03a46ced" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34155 } }
I20260807 22:45:39.867316 21602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:39.867347 21602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:39.867390 21602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:39.868983 21602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2256d458b2734c8d8075801a03a46ced" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34155 } }
I20260807 22:45:39.869189 21602 leader_election.cc:302] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2256d458b2734c8d8075801a03a46ced; no voters:
I20260807 22:45:39.869654 21602 leader_election.cc:288] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:39.869899 21607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:39.870007 21602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:39.870078 21607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [term 1 LEADER]: Becoming Leader. State: Replica: 2256d458b2734c8d8075801a03a46ced, State: Running, Role: LEADER
I20260807 22:45:39.870190 21607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2256d458b2734c8d8075801a03a46ced" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34155 } }
I20260807 22:45:39.870671 21608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2256d458b2734c8d8075801a03a46ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2256d458b2734c8d8075801a03a46ced" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34155 } } }
I20260807 22:45:39.870780 21609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2256d458b2734c8d8075801a03a46ced. Latest consensus state: current_term: 1 leader_uuid: "2256d458b2734c8d8075801a03a46ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2256d458b2734c8d8075801a03a46ced" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34155 } } }
I20260807 22:45:39.870837 21608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:39.870870 21609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:39.871163 21615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:39.871785 21615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:39.873680 21615 catalog_manager.cc:1347] Generated new cluster ID: 2292c4a0f8ae4cd4aea804dc15f54a79
I20260807 22:45:39.873745 21615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:45:39.886297 21615 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:39.886519 21615 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:39.892397 21615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced: Generated new TSK 0
I20260807 22:45:39.892688 21615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:45:39.897902 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34155
--builtin_ntp_servers=127.16.98.148:34825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 22:45:40.009085 21626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:40.009367 21626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:40.009400 21626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:40.012475 21626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:40.012637 21626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:40.012693 21626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:45:40.015677 21626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:40.016115 21626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:40.016530 21626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:40.023990 21631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:40.024149 21632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:40.024264 21634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:40.024771 21626 server_base.cc:1034] running on GCE node
I20260807 22:45:40.025063 21626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:40.025403 21626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:40.026614 21626 hybrid_clock.cc:648] HybridClock initialized: now 1786142740026589 us; error 47 us; skew 500 ppm
I20260807 22:45:40.028545 21626 webserver.cc:460] Webserver started at http://127.16.98.129:34939/ using document root <none> and password file <none>
I20260807 22:45:40.029377 21626 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:40.029719 21626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:40.029953 21626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:40.031311 21626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "65070385f468431984b6f7bf0a48288a"
format_stamp: "Formatted at 2026-08-07 22:45:40 on dist-test-slave-rgb1"
server_key: "efb07897c816101e6d15adc357080fcc"
server_key_iv: "afea96cd0118a0b7683554add8cb3942"
server_key_version: "encryptionkey@0"
I20260807 22:45:40.031751 21626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "65070385f468431984b6f7bf0a48288a"
format_stamp: "Formatted at 2026-08-07 22:45:40 on dist-test-slave-rgb1"
server_key: "efb07897c816101e6d15adc357080fcc"
server_key_iv: "afea96cd0118a0b7683554add8cb3942"
server_key_version: "encryptionkey@0"
I20260807 22:45:40.033587 21626 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 22:45:40.034965 21640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:40.035391 21626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:40.035578 21626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "65070385f468431984b6f7bf0a48288a"
format_stamp: "Formatted at 2026-08-07 22:45:40 on dist-test-slave-rgb1"
server_key: "efb07897c816101e6d15adc357080fcc"
server_key_iv: "afea96cd0118a0b7683554add8cb3942"
server_key_version: "encryptionkey@0"
I20260807 22:45:40.035706 21626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:40.040004 21626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:40.040418 21626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:40.040656 21626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:40.040988 21626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:40.041401 21626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:40.041468 21626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:40.041523 21626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:40.041560 21626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:40.049046 21626 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:40705
I20260807 22:45:40.049355 21757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:40705 every 8 connection(s)
I20260807 22:45:40.049625 21626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:45:40.054133 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.050644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:60267 (local address 127.16.98.190:34155)
0807 22:45:40.050828 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:40.050831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:40.050995 (+ 164us) server_negotiation.cc:407] Connection header received
0807 22:45:40.052018 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:40.052019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:40.052027 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:40.052083 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:40.053774 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:40.053777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:40.053777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:40.053783 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:40.053801 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:40.053806 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:40.053900 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:40.054020 (+ 120us) server_negotiation.cc:299] Negotiation successful
0807 22:45:40.054039 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260807 22:45:40.054474 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.050597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:34155 (local address 127.16.98.129:60267)
0807 22:45:40.050834 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:40.050876 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:40.051632 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:40.052184 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:40.052189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:40.052206 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:40.053577 (+ 1371us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:45:40.053577 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:40.053613 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:45:40.053649 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:40.053903 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:40.053912 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:40.053915 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:45:40.054035 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 22:45:40.054205 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260807 22:45:40.055115 21758 heartbeater.cc:347] Connected to a master server at 127.16.98.190:34155
I20260807 22:45:40.055227 21758 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:40.055377 21758 heartbeater.cc:511] Master 127.16.98.190:34155 requested a full tablet report, sending...
I20260807 22:45:40.055804 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 21626
I20260807 22:45:40.055946 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:45:40.056133 21564 ts_manager.cc:194] Registered new tserver with Master: 65070385f468431984b6f7bf0a48288a (127.16.98.129:40705)
I20260807 22:45:40.056190 16778 external_mini_cluster.cc:1428] Setting key c59a52bde23c3a34473f87e97d2225e6
I20260807 22:45:40.058528 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34155
--builtin_ntp_servers=127.16.98.148:34825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 22:45:40.164813 21761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:40.165078 21761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:40.165109 21761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:40.166599 21761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:40.166651 21761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:40.166684 21761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:45:40.168484 21761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:40.168855 21761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:40.169252 21761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:40.176589 21766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:40.176795 21769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:40.176888 21767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:40.177580 21761 server_base.cc:1034] running on GCE node
I20260807 22:45:40.177803 21761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:40.178076 21761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:40.179248 21761 hybrid_clock.cc:648] HybridClock initialized: now 1786142740179231 us; error 39 us; skew 500 ppm
I20260807 22:45:40.181301 21761 webserver.cc:460] Webserver started at http://127.16.98.130:37085/ using document root <none> and password file <none>
I20260807 22:45:40.181629 21761 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:40.181689 21761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:40.181846 21761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:40.183097 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "c2ffadddc38b49a49bd66442b3c3bb31"
format_stamp: "Formatted at 2026-08-07 22:45:40 on dist-test-slave-rgb1"
server_key: "d4daa4744595048667339694517d6f53"
server_key_iv: "774a9fddcc79966c91de9538f41fc12d"
server_key_version: "encryptionkey@0"
I20260807 22:45:40.183516 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "c2ffadddc38b49a49bd66442b3c3bb31"
format_stamp: "Formatted at 2026-08-07 22:45:40 on dist-test-slave-rgb1"
server_key: "d4daa4744595048667339694517d6f53"
server_key_iv: "774a9fddcc79966c91de9538f41fc12d"
server_key_version: "encryptionkey@0"
I20260807 22:45:40.185264 21761 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260807 22:45:40.186354 21775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:40.186661 21761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:40.186743 21761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "c2ffadddc38b49a49bd66442b3c3bb31"
format_stamp: "Formatted at 2026-08-07 22:45:40 on dist-test-slave-rgb1"
server_key: "d4daa4744595048667339694517d6f53"
server_key_iv: "774a9fddcc79966c91de9538f41fc12d"
server_key_version: "encryptionkey@0"
I20260807 22:45:40.186848 21761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:40.190627 21761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:40.191151 21761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:40.191325 21761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:40.191623 21761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:40.192010 21761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:40.192072 21761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:40.192119 21761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:40.192150 21761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:40.199549 21761 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:33981
I20260807 22:45:40.199595 21891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:33981 every 8 connection(s)
I20260807 22:45:40.200162 21761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:45:40.204404 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.201129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:34155 (local address 127.16.98.130:51865)
0807 22:45:40.201371 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:40.201417 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:40.201928 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:40.202450 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:40.202456 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:40.202473 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:40.203641 (+ 1168us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:45:40.203641 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:40.203671 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:45:40.203702 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:40.204056 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:40.204064 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:40.204067 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:45:40.204155 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 22:45:40.204223 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":112,"threads_started":1}
I20260807 22:45:40.204413 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.201223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:51865 (local address 127.16.98.190:34155)
0807 22:45:40.201321 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:40.201324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:40.201565 (+ 241us) server_negotiation.cc:407] Connection header received
0807 22:45:40.202131 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:40.202132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:40.202138 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:40.202208 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:40.203849 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:40.203851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:40.203851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:40.203856 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:40.203873 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:40.203877 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:40.203951 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:40.204337 (+ 386us) server_negotiation.cc:299] Negotiation successful
0807 22:45:40.204353 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 22:45:40.205235 21892 heartbeater.cc:347] Connected to a master server at 127.16.98.190:34155
I20260807 22:45:40.205382 21892 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:40.205668 21892 heartbeater.cc:511] Master 127.16.98.190:34155 requested a full tablet report, sending...
I20260807 22:45:40.206183 21564 ts_manager.cc:194] Registered new tserver with Master: c2ffadddc38b49a49bd66442b3c3bb31 (127.16.98.130:33981)
I20260807 22:45:40.207157 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 21761
I20260807 22:45:40.207268 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:45:40.207501 16778 external_mini_cluster.cc:1428] Setting key fef08e5e6fbf2eac4d19bcbe7b574579
I20260807 22:45:40.208758 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34155
--builtin_ntp_servers=127.16.98.148:34825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 22:45:40.300511 21895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:40.300765 21895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:40.300791 21895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:40.302060 21895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:40.302107 21895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:40.302136 21895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:45:40.303558 21895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:40.303835 21895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:40.304183 21895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:40.309649 21901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:40.309652 21903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:40.309667 21900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:40.309791 21895 server_base.cc:1034] running on GCE node
I20260807 22:45:40.310395 21895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:40.310992 21895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:40.312240 21895 hybrid_clock.cc:648] HybridClock initialized: now 1786142740312196 us; error 94 us; skew 500 ppm
I20260807 22:45:40.313737 21895 webserver.cc:460] Webserver started at http://127.16.98.131:41187/ using document root <none> and password file <none>
I20260807 22:45:40.314016 21895 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:40.314069 21895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:40.314215 21895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:40.315469 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "aaa24c5a63234a19add7cae9172bf6da"
format_stamp: "Formatted at 2026-08-07 22:45:40 on dist-test-slave-rgb1"
server_key: "67de02b217183960c31ae708f3ce701b"
server_key_iv: "bc6590d8a28b0291ec86ed10fead3f5a"
server_key_version: "encryptionkey@0"
I20260807 22:45:40.315841 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "aaa24c5a63234a19add7cae9172bf6da"
format_stamp: "Formatted at 2026-08-07 22:45:40 on dist-test-slave-rgb1"
server_key: "67de02b217183960c31ae708f3ce701b"
server_key_iv: "bc6590d8a28b0291ec86ed10fead3f5a"
server_key_version: "encryptionkey@0"
I20260807 22:45:40.318161 21895 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260807 22:45:40.319193 21909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:40.319479 21895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:40.319599 21895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "aaa24c5a63234a19add7cae9172bf6da"
format_stamp: "Formatted at 2026-08-07 22:45:40 on dist-test-slave-rgb1"
server_key: "67de02b217183960c31ae708f3ce701b"
server_key_iv: "bc6590d8a28b0291ec86ed10fead3f5a"
server_key_version: "encryptionkey@0"
I20260807 22:45:40.319700 21895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:40.323772 21895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:40.324079 21895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:40.324246 21895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:40.324474 21895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:40.324831 21895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:40.324896 21895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:40.324939 21895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:40.324967 21895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:40.331457 21895 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:39991
I20260807 22:45:40.331506 22026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:39991 every 8 connection(s)
I20260807 22:45:40.331871 21895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:45:40.335739 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 21895
I20260807 22:45:40.335757 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.332767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:37179 (local address 127.16.98.190:34155)
0807 22:45:40.332857 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:40.332860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:40.333117 (+ 257us) server_negotiation.cc:407] Connection header received
0807 22:45:40.333729 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:40.333731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:40.333738 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:40.333818 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:40.335340 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:40.335342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:40.335342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:40.335349 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:40.335368 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:40.335373 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:40.335477 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:40.335645 (+ 168us) server_negotiation.cc:299] Negotiation successful
0807 22:45:40.335666 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 22:45:40.335897 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:45:40.336007 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.332639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:34155 (local address 127.16.98.131:37179)
0807 22:45:40.332954 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:40.333009 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:40.333599 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:40.333915 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:40.333919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:40.333939 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:40.335135 (+ 1196us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:45:40.335136 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:40.335170 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:45:40.335205 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:40.335480 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:40.335517 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:40.335524 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 22:45:40.335661 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 22:45:40.335739 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":109,"threads_started":1}
I20260807 22:45:40.336104 16778 external_mini_cluster.cc:1428] Setting key 4df428983d32134ae930cd22d9e45a31
I20260807 22:45:40.336710 22027 heartbeater.cc:347] Connected to a master server at 127.16.98.190:34155
I20260807 22:45:40.336831 22027 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:40.336998 22027 heartbeater.cc:511] Master 127.16.98.190:34155 requested a full tablet report, sending...
I20260807 22:45:40.337574 21563 ts_manager.cc:194] Registered new tserver with Master: aaa24c5a63234a19add7cae9172bf6da (127.16.98.131:39991)
I20260807 22:45:40.348903 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:45:40.352355 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.350661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38512 (local address 127.16.98.190:34155)
0807 22:45:40.350734 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:40.350736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:40.350828 (+ 92us) server_negotiation.cc:407] Connection header received
0807 22:45:40.350909 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:40.350910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:40.350915 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:40.351069 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:40.351838 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:40.351839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:40.351840 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:40.351844 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:40.351863 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:40.351874 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:40.352059 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:40.352239 (+ 180us) server_negotiation.cc:299] Negotiation successful
0807 22:45:40.352263 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 22:45:40.355336 21563 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:45:40.357422 21563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:40.363440 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.361171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.16.98.129:40705)
0807 22:45:40.361431 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:40.361437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:40.361451 (+ 14us) server_negotiation.cc:407] Connection header received
0807 22:45:40.361522 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:40.361526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:40.361537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:40.361630 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:40.362950 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:40.362959 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:40.362959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:40.362965 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:40.363005 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:40.363031 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:40.363083 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:40.363284 (+ 201us) server_negotiation.cc:299] Negotiation successful
0807 22:45:40.363306 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":121,"threads_started":1}
I20260807 22:45:40.363714 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.361063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:40705 (local address 127.0.0.1:33718)
0807 22:45:40.361320 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:40.361372 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:40.361450 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:40.361998 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:40.362006 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:40.362015 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:40.362758 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:45:40.362759 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:40.362779 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:45:40.362818 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:40.363148 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:40.363156 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:40.363159 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:45:40.363593 (+ 434us) client_negotiation.cc:240] Negotiation successful
0807 22:45:40.363616 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":106,"threads_started":1}
I20260807 22:45:40.364069 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.361156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:33981 (local address 127.0.0.1:39124)
0807 22:45:40.361720 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:40.361735 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:40.361819 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:40.362234 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:40.362235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:40.362238 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:40.363459 (+ 1221us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:45:40.363460 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:40.363468 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:45:40.363493 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:40.363931 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:40.363932 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:40.363933 (+ 1us) client_negotiation.cc:769] Sending connection context
0807 22:45:40.363968 (+ 35us) client_negotiation.cc:240] Negotiation successful
0807 22:45:40.363982 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":132,"threads_started":1}
I20260807 22:45:40.364348 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.361560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.16.98.130:33981)
0807 22:45:40.361973 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:40.361979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:40.361989 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:45:40.362044 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:40.362047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:40.362061 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:40.362153 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:40.363701 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:40.363720 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:40.363720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:40.363728 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:40.363777 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:40.363827 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:40.364201 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:40.364231 (+ 30us) server_negotiation.cc:299] Negotiation successful
0807 22:45:40.364253 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":115,"threads_started":1}
I20260807 22:45:40.364744 21691 tablet_service.cc:1467] Processing CreateTablet for tablet 9e4f39f19b74495594593397ad7f0f01 (DEFAULT_TABLE table=test-table [id=cb23088a8dcc4204a9c225a6eee7aaf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:40.365041 22040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.361643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.16.98.131:39991)
0807 22:45:40.362762 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:40.362767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:40.362782 (+ 15us) server_negotiation.cc:407] Connection header received
0807 22:45:40.362815 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:40.362817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:40.362828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:40.362925 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:40.364649 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:40.364660 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:40.364661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:40.364668 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:40.364711 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:40.364739 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:40.364822 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:40.364923 (+ 101us) server_negotiation.cc:299] Negotiation successful
0807 22:45:40.364943 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068,"thread_start_us":136,"threads_started":1}
I20260807 22:45:40.365074 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.361291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:39991 (local address 127.0.0.1:44456)
0807 22:45:40.361805 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:40.361815 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:40.361863 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:40.363250 (+ 1387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:40.363251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:40.363255 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:40.364496 (+ 1241us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:45:40.364498 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:40.364505 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:45:40.364520 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:40.364830 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:40.364831 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:40.364832 (+ 1us) client_negotiation.cc:769] Sending connection context
0807 22:45:40.364896 (+ 64us) client_negotiation.cc:240] Negotiation successful
0807 22:45:40.364911 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":113,"threads_started":1}
I20260807 22:45:40.365214 21691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e4f39f19b74495594593397ad7f0f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:40.365164 21824 tablet_service.cc:1467] Processing CreateTablet for tablet 9e4f39f19b74495594593397ad7f0f01 (DEFAULT_TABLE table=test-table [id=cb23088a8dcc4204a9c225a6eee7aaf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:40.365552 21824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e4f39f19b74495594593397ad7f0f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:40.365778 21958 tablet_service.cc:1467] Processing CreateTablet for tablet 9e4f39f19b74495594593397ad7f0f01 (DEFAULT_TABLE table=test-table [id=cb23088a8dcc4204a9c225a6eee7aaf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:40.366088 21958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e4f39f19b74495594593397ad7f0f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:40.368497 22042 tablet_bootstrap.cc:492] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31: Bootstrap starting.
I20260807 22:45:40.369511 22042 tablet_bootstrap.cc:654] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:40.369858 22042 log.cc:824] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:40.369692 22043 tablet_bootstrap.cc:492] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a: Bootstrap starting.
I20260807 22:45:40.370838 22042 tablet_bootstrap.cc:492] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31: No bootstrap required, opened a new log
I20260807 22:45:40.370945 22042 ts_tablet_manager.cc:1397] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:45:40.371330 22043 tablet_bootstrap.cc:654] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:40.371796 22045 tablet_bootstrap.cc:492] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da: Bootstrap starting.
I20260807 22:45:40.372062 22043 log.cc:824] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:40.372911 22045 tablet_bootstrap.cc:654] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:40.373369 22043 tablet_bootstrap.cc:492] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a: No bootstrap required, opened a new log
I20260807 22:45:40.373457 22043 ts_tablet_manager.cc:1397] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260807 22:45:40.374073 22045 log.cc:824] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:40.375089 22042 raft_consensus.cc:348] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ffadddc38b49a49bd66442b3c3bb31" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 33981 } } peers { permanent_uuid: "65070385f468431984b6f7bf0a48288a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40705 } } peers { permanent_uuid: "aaa24c5a63234a19add7cae9172bf6da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39991 } }
I20260807 22:45:40.375299 22042 raft_consensus.cc:374] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:40.375346 22042 raft_consensus.cc:729] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ffadddc38b49a49bd66442b3c3bb31, State: Initialized, Role: FOLLOWER
I20260807 22:45:40.375387 22045 tablet_bootstrap.cc:492] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da: No bootstrap required, opened a new log
I20260807 22:45:40.375452 22045 ts_tablet_manager.cc:1397] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260807 22:45:40.375519 22042 consensus_queue.cc:260] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ffadddc38b49a49bd66442b3c3bb31" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 33981 } } peers { permanent_uuid: "65070385f468431984b6f7bf0a48288a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40705 } } peers { permanent_uuid: "aaa24c5a63234a19add7cae9172bf6da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39991 } }
I20260807 22:45:40.375856 22042 ts_tablet_manager.cc:1428] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 22:45:40.375984 21892 heartbeater.cc:503] Master 127.16.98.190:34155 was elected leader, sending a full tablet report...
I20260807 22:45:40.377269 22043 raft_consensus.cc:348] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ffadddc38b49a49bd66442b3c3bb31" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 33981 } } peers { permanent_uuid: "65070385f468431984b6f7bf0a48288a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40705 } } peers { permanent_uuid: "aaa24c5a63234a19add7cae9172bf6da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39991 } }
I20260807 22:45:40.377545 22043 raft_consensus.cc:374] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:40.377621 22043 raft_consensus.cc:729] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65070385f468431984b6f7bf0a48288a, State: Initialized, Role: FOLLOWER
I20260807 22:45:40.377774 22043 consensus_queue.cc:260] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ffadddc38b49a49bd66442b3c3bb31" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 33981 } } peers { permanent_uuid: "65070385f468431984b6f7bf0a48288a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40705 } } peers { permanent_uuid: "aaa24c5a63234a19add7cae9172bf6da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39991 } }
I20260807 22:45:40.378047 22043 ts_tablet_manager.cc:1428] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 22:45:40.378147 21758 heartbeater.cc:503] Master 127.16.98.190:34155 was elected leader, sending a full tablet report...
I20260807 22:45:40.378403 22045 raft_consensus.cc:348] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ffadddc38b49a49bd66442b3c3bb31" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 33981 } } peers { permanent_uuid: "65070385f468431984b6f7bf0a48288a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40705 } } peers { permanent_uuid: "aaa24c5a63234a19add7cae9172bf6da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39991 } }
I20260807 22:45:40.378597 22045 raft_consensus.cc:374] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:40.378649 22045 raft_consensus.cc:729] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa24c5a63234a19add7cae9172bf6da, State: Initialized, Role: FOLLOWER
I20260807 22:45:40.378808 22045 consensus_queue.cc:260] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ffadddc38b49a49bd66442b3c3bb31" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 33981 } } peers { permanent_uuid: "65070385f468431984b6f7bf0a48288a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40705 } } peers { permanent_uuid: "aaa24c5a63234a19add7cae9172bf6da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39991 } }
I20260807 22:45:40.379110 22045 ts_tablet_manager.cc:1428] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260807 22:45:40.379154 22027 heartbeater.cc:503] Master 127.16.98.190:34155 was elected leader, sending a full tablet report...
W20260807 22:45:40.450992 21893 tablet.cc:2369] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:45:40.550745 21759 tablet.cc:2369] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:45:40.582557 22028 tablet.cc:2369] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:45:40.609869 22049 raft_consensus.cc:482] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:40.610024 22049 raft_consensus.cc:504] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ffadddc38b49a49bd66442b3c3bb31" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 33981 } } peers { permanent_uuid: "65070385f468431984b6f7bf0a48288a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40705 } } peers { permanent_uuid: "aaa24c5a63234a19add7cae9172bf6da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39991 } }
I20260807 22:45:40.610566 22049 leader_election.cc:288] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ffadddc38b49a49bd66442b3c3bb31 (127.16.98.130:33981), aaa24c5a63234a19add7cae9172bf6da (127.16.98.131:39991)
I20260807 22:45:40.612232 22052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.610800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:33981 (local address 127.16.98.129:49653)
0807 22:45:40.611002 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:40.611017 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:40.611089 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:40.611245 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:40.611246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:40.611249 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:40.611795 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:45:40.611795 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:40.611804 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:45:40.611819 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:40.612058 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:40.612059 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:40.612060 (+ 1us) client_negotiation.cc:769] Sending connection context
0807 22:45:40.612106 (+ 46us) client_negotiation.cc:240] Negotiation successful
0807 22:45:40.612118 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":109,"threads_started":1}
I20260807 22:45:40.612221 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.610949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:49653 (local address 127.16.98.130:33981)
0807 22:45:40.611017 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:40.611019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:40.611107 (+ 88us) server_negotiation.cc:407] Connection header received
0807 22:45:40.611148 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:40.611149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:40.611156 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:40.611228 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:40.611939 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:40.611940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:40.611941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:40.611946 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:40.611966 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:40.611971 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:40.612057 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:40.612107 (+ 50us) server_negotiation.cc:299] Negotiation successful
0807 22:45:40.612132 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 22:45:40.612506 22040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.610978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:42991 (local address 127.16.98.131:39991)
0807 22:45:40.611027 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:40.611029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:40.611229 (+ 200us) server_negotiation.cc:407] Connection header received
0807 22:45:40.611255 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:40.611256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:40.611261 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:40.611314 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:40.612184 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:40.612185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:40.612185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:40.612189 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:40.612204 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:40.612208 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:40.612267 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:40.612390 (+ 123us) server_negotiation.cc:299] Negotiation successful
0807 22:45:40.612407 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 22:45:40.612739 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.610886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:39991 (local address 127.16.98.129:42991)
0807 22:45:40.611101 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:40.611116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:40.611185 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:40.611343 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:40.611343 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:40.611361 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:40.612066 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:45:40.612066 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:40.612074 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:45:40.612090 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:40.612285 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:40.612286 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:40.612287 (+ 1us) client_negotiation.cc:769] Sending connection context
0807 22:45:40.612638 (+ 351us) client_negotiation.cc:240] Negotiation successful
0807 22:45:40.612656 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
I20260807 22:45:40.612776 21845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e4f39f19b74495594593397ad7f0f01" candidate_uuid: "65070385f468431984b6f7bf0a48288a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ffadddc38b49a49bd66442b3c3bb31" is_pre_election: true
I20260807 22:45:40.612939 21845 raft_consensus.cc:2454] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65070385f468431984b6f7bf0a48288a in term 0.
I20260807 22:45:40.613369 21979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e4f39f19b74495594593397ad7f0f01" candidate_uuid: "65070385f468431984b6f7bf0a48288a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa24c5a63234a19add7cae9172bf6da" is_pre_election: true
I20260807 22:45:40.613363 21641 leader_election.cc:302] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65070385f468431984b6f7bf0a48288a, c2ffadddc38b49a49bd66442b3c3bb31; no voters:
I20260807 22:45:40.613498 21979 raft_consensus.cc:2454] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65070385f468431984b6f7bf0a48288a in term 0.
I20260807 22:45:40.613607 22049 raft_consensus.cc:2781] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:40.613694 22049 raft_consensus.cc:482] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:40.613729 22049 raft_consensus.cc:3037] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:40.614907 22049 raft_consensus.cc:504] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ffadddc38b49a49bd66442b3c3bb31" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 33981 } } peers { permanent_uuid: "65070385f468431984b6f7bf0a48288a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40705 } } peers { permanent_uuid: "aaa24c5a63234a19add7cae9172bf6da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39991 } }
I20260807 22:45:40.615104 22049 leader_election.cc:288] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [CANDIDATE]: Term 1 election: Requested vote from peers c2ffadddc38b49a49bd66442b3c3bb31 (127.16.98.130:33981), aaa24c5a63234a19add7cae9172bf6da (127.16.98.131:39991)
I20260807 22:45:40.615322 21979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e4f39f19b74495594593397ad7f0f01" candidate_uuid: "65070385f468431984b6f7bf0a48288a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa24c5a63234a19add7cae9172bf6da"
I20260807 22:45:40.615330 21845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e4f39f19b74495594593397ad7f0f01" candidate_uuid: "65070385f468431984b6f7bf0a48288a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ffadddc38b49a49bd66442b3c3bb31"
I20260807 22:45:40.615414 21979 raft_consensus.cc:3037] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:40.615415 21845 raft_consensus.cc:3037] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:40.616360 21979 raft_consensus.cc:2454] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65070385f468431984b6f7bf0a48288a in term 1.
I20260807 22:45:40.616361 21845 raft_consensus.cc:2454] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65070385f468431984b6f7bf0a48288a in term 1.
I20260807 22:45:40.616641 21644 leader_election.cc:302] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65070385f468431984b6f7bf0a48288a, aaa24c5a63234a19add7cae9172bf6da; no voters:
I20260807 22:45:40.616792 22049 raft_consensus.cc:2781] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:40.616966 22049 raft_consensus.cc:686] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [term 1 LEADER]: Becoming Leader. State: Replica: 65070385f468431984b6f7bf0a48288a, State: Running, Role: LEADER
I20260807 22:45:40.617151 22049 consensus_queue.cc:237] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ffadddc38b49a49bd66442b3c3bb31" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 33981 } } peers { permanent_uuid: "65070385f468431984b6f7bf0a48288a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40705 } } peers { permanent_uuid: "aaa24c5a63234a19add7cae9172bf6da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39991 } }
I20260807 22:45:40.618049 21564 catalog_manager.cc:5495] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a reported cstate change: term changed from 0 to 1, leader changed from <none> to 65070385f468431984b6f7bf0a48288a (127.16.98.129). New cstate: current_term: 1 leader_uuid: "65070385f468431984b6f7bf0a48288a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ffadddc38b49a49bd66442b3c3bb31" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 33981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65070385f468431984b6f7bf0a48288a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa24c5a63234a19add7cae9172bf6da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39991 } health_report { overall_health: UNKNOWN } } }
I20260807 22:45:40.663859 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.662406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.16.98.129:40705)
0807 22:45:40.662525 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:40.662527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:40.662540 (+ 13us) server_negotiation.cc:407] Connection header received
0807 22:45:40.662628 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:40.662631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:40.662656 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:40.662756 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:40.663586 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:40.663588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:40.663588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:40.663593 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:40.663616 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:40.663620 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:40.663690 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:40.663758 (+ 68us) server_negotiation.cc:299] Negotiation successful
0807 22:45:40.663772 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 22:45:40.666311 21845 raft_consensus.cc:1261] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31 [term 1 FOLLOWER]: Refusing update from remote peer 65070385f468431984b6f7bf0a48288a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:40.666373 21979 raft_consensus.cc:1261] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da [term 1 FOLLOWER]: Refusing update from remote peer 65070385f468431984b6f7bf0a48288a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:40.666867 22053 consensus_queue.cc:1035] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ffadddc38b49a49bd66442b3c3bb31" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:40.667176 22049 consensus_queue.cc:1035] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa24c5a63234a19add7cae9172bf6da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:40.671736 22057 mvcc.cc:205] Tried to move back new op lower bound from 7316040665765974016 to 7316040665568587776. Current Snapshot: MvccSnapshot[applied={T|T < 7316040665765974016}]
I20260807 22:45:40.672338 22059 mvcc.cc:205] Tried to move back new op lower bound from 7316040665765974016 to 7316040665568587776. Current Snapshot: MvccSnapshot[applied={T|T < 7316040665765974016}]
I20260807 22:45:40.675271 21564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:40.675421 21564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:40.676447 21564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2256d458b2734c8d8075801a03a46ced: Sending DeleteTablet for 3 replicas of tablet 9e4f39f19b74495594593397ad7f0f01
I20260807 22:45:40.676913 21691 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e4f39f19b74495594593397ad7f0f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:40 UTC) from {username='slave'} at 127.0.0.1:33718
I20260807 22:45:40.677003 21958 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e4f39f19b74495594593397ad7f0f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:40 UTC) from {username='slave'} at 127.0.0.1:44456
I20260807 22:45:40.677026 21824 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e4f39f19b74495594593397ad7f0f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:40 UTC) from {username='slave'} at 127.0.0.1:39124
I20260807 22:45:40.677294 22068 tablet_replica.cc:330] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da: stopping tablet replica
I20260807 22:45:40.677424 22066 tablet_replica.cc:330] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a: stopping tablet replica
I20260807 22:45:40.677454 22068 raft_consensus.cc:2229] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:40.677546 22066 raft_consensus.cc:2229] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:45:40.677594 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 21626
I20260807 22:45:40.677668 22066 raft_consensus.cc:2258] T 9e4f39f19b74495594593397ad7f0f01 P 65070385f468431984b6f7bf0a48288a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:40.677788 22068 raft_consensus.cc:2258] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:40.677992 22067 tablet_replica.cc:330] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31: stopping tablet replica
I20260807 22:45:40.678226 22067 raft_consensus.cc:2229] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:40.678362 22067 raft_consensus.cc:2258] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:40.678939 22067 ts_tablet_manager.cc:1905] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:40.680061 22068 ts_tablet_manager.cc:1905] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:40.681502 22067 ts_tablet_manager.cc:1918] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:40.681694 22067 log.cc:1196] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/9e4f39f19b74495594593397ad7f0f01
I20260807 22:45:40.682116 22067 ts_tablet_manager.cc:1939] T 9e4f39f19b74495594593397ad7f0f01 P c2ffadddc38b49a49bd66442b3c3bb31: Deleting consensus metadata
I20260807 22:45:40.682195 22068 ts_tablet_manager.cc:1918] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:40.682281 22068 log.cc:1196] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/9e4f39f19b74495594593397ad7f0f01
I20260807 22:45:40.682554 22068 ts_tablet_manager.cc:1939] T 9e4f39f19b74495594593397ad7f0f01 P aaa24c5a63234a19add7cae9172bf6da: Deleting consensus metadata
I20260807 22:45:40.683000 21548 catalog_manager.cc:4841] TS c2ffadddc38b49a49bd66442b3c3bb31 (127.16.98.130:33981): tablet 9e4f39f19b74495594593397ad7f0f01 (table test-table [id=cb23088a8dcc4204a9c225a6eee7aaf9]) successfully deleted
I20260807 22:45:40.683779 21550 catalog_manager.cc:4841] TS aaa24c5a63234a19add7cae9172bf6da (127.16.98.131:39991): tablet 9e4f39f19b74495594593397ad7f0f01 (table test-table [id=cb23088a8dcc4204a9c225a6eee7aaf9]) successfully deleted
W20260807 22:45:40.688673 21551 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:40705 (error 108)
I20260807 22:45:40.688818 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 21761
I20260807 22:45:40.689991 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.689642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:40705 (local address 127.0.0.1:33734)
0807 22:45:40.689865 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:40.689929 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
W20260807 22:45:40.690374 21551 catalog_manager.cc:4584] TS 65070385f468431984b6f7bf0a48288a (127.16.98.129:40705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e4f39f19b74495594593397ad7f0f01: Network error: Client connection negotiation failed: client connection to 127.16.98.129:40705: connect: Connection refused (error 111)
I20260807 22:45:40.695784 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 21895
I20260807 22:45:40.701784 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 21533
2026-08-07T22:45:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (999 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260807 22:45:40.725463 16778 mini_oidc.cc:152] Starting JWKS server
I20260807 22:45:40.726719 16778 webserver.cc:460] Webserver started at https://127.0.0.1:33043/ using document root <none> and password file <none>
I20260807 22:45:40.726854 16778 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 22:45:40.727066 16778 webserver.cc:460] Webserver started at http://127.0.0.1:33285/ using document root <none> and password file <none>
2026-08-07T22:45:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:40Z Disabled control of system clock
I20260807 22:45:40.739753 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:40023
--webserver_interface=127.16.98.190
--webserver_port=0
--builtin_ntp_servers=127.16.98.148:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:40023
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33043/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/testchainca.cert with env {}
W20260807 22:45:40.832783 22075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:40.833020 22075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:40.833045 22075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:40.834281 22075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:40.834331 22075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:40.834343 22075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:40.834357 22075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:40.834369 22075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 22:45:40.834383 22075 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 22:45:40.834442 22075 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33043/jwks.json
I20260807 22:45:40.836180 22075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:40023
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:40023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33043/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:40.836481 22075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:40.836756 22075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:40.842090 22083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:40.842204 22081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:40.842336 22075 server_base.cc:1034] running on GCE node
W20260807 22:45:40.842227 22080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:40.842751 22075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:40.843106 22075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:40.844312 22075 hybrid_clock.cc:648] HybridClock initialized: now 1786142740844288 us; error 44 us; skew 500 ppm
I20260807 22:45:40.846320 22075 webserver.cc:460] Webserver started at http://127.16.98.190:41469/ using document root <none> and password file <none>
I20260807 22:45:40.846635 22075 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:40.846721 22075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:40.846906 22075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:40.848143 22075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "9252d5fcfb354d568a0b35c76b5b2b7a"
format_stamp: "Formatted at 2026-08-07 22:45:40 on dist-test-slave-rgb1"
server_key: "b504960446e86c91852a38a19f23f8f6"
server_key_iv: "5dd7c87086213deaa91014c8c02215e6"
server_key_version: "encryptionkey@0"
I20260807 22:45:40.849062 22075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "9252d5fcfb354d568a0b35c76b5b2b7a"
format_stamp: "Formatted at 2026-08-07 22:45:40 on dist-test-slave-rgb1"
server_key: "b504960446e86c91852a38a19f23f8f6"
server_key_iv: "5dd7c87086213deaa91014c8c02215e6"
server_key_version: "encryptionkey@0"
I20260807 22:45:40.851795 22075 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260807 22:45:40.853109 22089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:40.853364 22075 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260807 22:45:40.853471 22075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "9252d5fcfb354d568a0b35c76b5b2b7a"
format_stamp: "Formatted at 2026-08-07 22:45:40 on dist-test-slave-rgb1"
server_key: "b504960446e86c91852a38a19f23f8f6"
server_key_iv: "5dd7c87086213deaa91014c8c02215e6"
server_key_version: "encryptionkey@0"
I20260807 22:45:40.853577 22075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:40.878290 22075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:40.878693 22075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:40.878852 22075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:40.883635 22075 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:40023
I20260807 22:45:40.883702 22144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:40023 every 8 connection(s)
I20260807 22:45:40.884043 22075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:40.885605 22145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:40.888388 22145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a: Bootstrap starting.
I20260807 22:45:40.889170 22145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:40.889467 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 22075
I20260807 22:45:40.889608 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:40.889603 22145 log.cc:824] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:40.889830 16778 external_mini_cluster.cc:1428] Setting key 9f2ebc2e6cc246bbaf00128bb509d2dc
I20260807 22:45:40.890684 22145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a: No bootstrap required, opened a new log
I20260807 22:45:40.892750 22145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9252d5fcfb354d568a0b35c76b5b2b7a" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40023 } }
I20260807 22:45:40.892951 22145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:40.893018 22145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9252d5fcfb354d568a0b35c76b5b2b7a, State: Initialized, Role: FOLLOWER
I20260807 22:45:40.893234 22145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9252d5fcfb354d568a0b35c76b5b2b7a" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40023 } }
I20260807 22:45:40.893347 22145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:40.893394 22145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:40.893440 22145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:40.894395 22145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9252d5fcfb354d568a0b35c76b5b2b7a" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40023 } }
I20260807 22:45:40.894526 22145 leader_election.cc:302] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9252d5fcfb354d568a0b35c76b5b2b7a; no voters:
I20260807 22:45:40.894711 22145 leader_election.cc:288] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:40.894945 22150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:40.895115 22145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:40.895259 22150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [term 1 LEADER]: Becoming Leader. State: Replica: 9252d5fcfb354d568a0b35c76b5b2b7a, State: Running, Role: LEADER
I20260807 22:45:40.895406 22150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9252d5fcfb354d568a0b35c76b5b2b7a" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40023 } }
I20260807 22:45:40.896111 22150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9252d5fcfb354d568a0b35c76b5b2b7a. Latest consensus state: current_term: 1 leader_uuid: "9252d5fcfb354d568a0b35c76b5b2b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9252d5fcfb354d568a0b35c76b5b2b7a" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40023 } } }
I20260807 22:45:40.896440 22150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:40.896481 22151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9252d5fcfb354d568a0b35c76b5b2b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9252d5fcfb354d568a0b35c76b5b2b7a" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40023 } } }
I20260807 22:45:40.896791 22163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:40.896780 22151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:40.897709 22163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:40.899689 22163 catalog_manager.cc:1347] Generated new cluster ID: 055c33d73d1247ccbff04105fed81d53
I20260807 22:45:40.899756 22163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:45:40.900041 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.890495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51934 (local address 127.16.98.190:40023)
0807 22:45:40.890819 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:40.890830 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:40.890878 (+ 48us) server_negotiation.cc:407] Connection header received
0807 22:45:40.892767 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:40.892787 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:40.892923 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:40.893256 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:40.893695 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:40.895737 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:40.897243 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:40.897940 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:40.898084 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:40.899046 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:40.899050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:40.899051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:40.899304 (+ 253us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:40.899357 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:40.899375 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:40.899675 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:40.899713 (+ 38us) server_negotiation.cc:299] Negotiation successful
0807 22:45:40.899807 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
I20260807 22:45:40.901212 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:45:40.930467 22163 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:40.931114 22163 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:40.938126 22163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9252d5fcfb354d568a0b35c76b5b2b7a: Generated new TSK 0
I20260807 22:45:40.938452 22163 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 22:45:40.972129 22176 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260807 22:45:40.972302 22176 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 22:45:40.972548 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:40.954352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51950 (local address 127.16.98.190:40023)
0807 22:45:40.954465 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:40.954467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:40.954598 (+ 131us) server_negotiation.cc:407] Connection header received
0807 22:45:40.954748 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:40.954750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:40.954762 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:40.954834 (+ 72us) server_negotiation.cc:226] Negotiated authn=JWT
0807 22:45:40.955051 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:40.955537 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:40.957033 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:40.957230 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:40.957296 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:40.957618 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 22:45:40.972268 (+ 14650us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 22:45:40.972467 (+ 199us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51950: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260807 22:45:40.972628 22149 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51950: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260807 22:45:40.973050 22175 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:40.954279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:40023 (local address 127.0.0.1:51950)
0807 22:45:40.954504 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:40.954518 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:40.954606 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:40.954863 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:40.954865 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:40.954867 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0807 22:45:40.954943 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:40.954946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:40.955757 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:40.955758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:40.956850 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:40.956854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:40.957332 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:40.957333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:40.957401 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:40.957432 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 22:45:40.972553 (+ 15121us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 22:45:40.972756 (+ 203us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.98.190:40023: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":120,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.98.190:40023: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260807 22:45:40.973537 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 22075
2026-08-07T22:45:40Z chronyd exiting
I20260807 22:45:41.012185 16778 test_util.cc:175] -----------------------------------------------
I20260807 22:45:41.012288 16778 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786142716408518-16778-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (288 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260807 22:45:41.013305 16778 mini_oidc.cc:152] Starting JWKS server
I20260807 22:45:41.014145 16778 webserver.cc:460] Webserver started at https://127.0.0.1:39517/ using document root <none> and password file <none>
I20260807 22:45:41.014228 16778 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 22:45:41.014406 16778 webserver.cc:460] Webserver started at http://127.0.0.1:40583/ using document root <none> and password file <none>
2026-08-07T22:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:41Z Disabled control of system clock
I20260807 22:45:41.025202 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:38009
--webserver_interface=127.16.98.190
--webserver_port=0
--builtin_ntp_servers=127.16.98.148:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:38009
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39517/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/testchainca.cert with env {}
W20260807 22:45:41.120141 22182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:41.120426 22182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:41.120453 22182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:41.121697 22182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:41.121744 22182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:41.121757 22182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:41.121770 22182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:41.121783 22182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 22:45:41.121795 22182 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 22:45:41.121806 22182 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39517/jwks.json
I20260807 22:45:41.123348 22182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:38009
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:38009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39517/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:41.123577 22182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:41.123749 22182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:41.128870 22188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:41.128877 22187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:41.128890 22190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:41.129223 22182 server_base.cc:1034] running on GCE node
I20260807 22:45:41.129518 22182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:41.129841 22182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:41.131018 22182 hybrid_clock.cc:648] HybridClock initialized: now 1786142741130997 us; error 46 us; skew 500 ppm
I20260807 22:45:41.132437 22182 webserver.cc:460] Webserver started at http://127.16.98.190:35159/ using document root <none> and password file <none>
I20260807 22:45:41.132740 22182 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:41.132818 22182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:41.132984 22182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:41.134387 22182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "44ded3dfdcb143349b9cfaf927edc1c8"
format_stamp: "Formatted at 2026-08-07 22:45:41 on dist-test-slave-rgb1"
server_key: "df77d57b0bd6f3d98f849a36c563e418"
server_key_iv: "b75ac9b8ffbe4da4bb29bdcc975d3410"
server_key_version: "encryptionkey@0"
I20260807 22:45:41.134816 22182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "44ded3dfdcb143349b9cfaf927edc1c8"
format_stamp: "Formatted at 2026-08-07 22:45:41 on dist-test-slave-rgb1"
server_key: "df77d57b0bd6f3d98f849a36c563e418"
server_key_iv: "b75ac9b8ffbe4da4bb29bdcc975d3410"
server_key_version: "encryptionkey@0"
I20260807 22:45:41.136269 22182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:45:41.137276 22196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:41.137497 22182 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 22:45:41.137611 22182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "44ded3dfdcb143349b9cfaf927edc1c8"
format_stamp: "Formatted at 2026-08-07 22:45:41 on dist-test-slave-rgb1"
server_key: "df77d57b0bd6f3d98f849a36c563e418"
server_key_iv: "b75ac9b8ffbe4da4bb29bdcc975d3410"
server_key_version: "encryptionkey@0"
I20260807 22:45:41.137704 22182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:41.175544 22182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:41.175961 22182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:41.176155 22182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:41.181565 22182 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:38009
I20260807 22:45:41.181650 22249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:38009 every 8 connection(s)
I20260807 22:45:41.182129 22182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:41.183331 22250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:41.186110 22250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8: Bootstrap starting.
I20260807 22:45:41.186730 22250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:41.187011 22250 log.cc:824] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:41.187842 22250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8: No bootstrap required, opened a new log
I20260807 22:45:41.189857 22250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ded3dfdcb143349b9cfaf927edc1c8" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38009 } }
I20260807 22:45:41.190030 22250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:41.190057 22250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ded3dfdcb143349b9cfaf927edc1c8, State: Initialized, Role: FOLLOWER
I20260807 22:45:41.190222 22250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ded3dfdcb143349b9cfaf927edc1c8" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38009 } }
I20260807 22:45:41.190315 22250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:41.190342 22250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:41.190374 22250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:41.190989 22250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ded3dfdcb143349b9cfaf927edc1c8" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38009 } }
I20260807 22:45:41.191097 22250 leader_election.cc:302] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44ded3dfdcb143349b9cfaf927edc1c8; no voters:
I20260807 22:45:41.191246 22250 leader_election.cc:288] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:41.191393 22253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:41.191603 22250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:41.191810 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 22182
I20260807 22:45:41.191859 22253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [term 1 LEADER]: Becoming Leader. State: Replica: 44ded3dfdcb143349b9cfaf927edc1c8, State: Running, Role: LEADER
I20260807 22:45:41.191928 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:41.192031 22253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ded3dfdcb143349b9cfaf927edc1c8" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38009 } }
I20260807 22:45:41.192202 16778 external_mini_cluster.cc:1428] Setting key f55dff5121fcd9f3a5aeb01cef49ce32
I20260807 22:45:41.192850 22254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44ded3dfdcb143349b9cfaf927edc1c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ded3dfdcb143349b9cfaf927edc1c8" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38009 } } }
I20260807 22:45:41.192970 22254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:41.193243 22255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44ded3dfdcb143349b9cfaf927edc1c8. Latest consensus state: current_term: 1 leader_uuid: "44ded3dfdcb143349b9cfaf927edc1c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ded3dfdcb143349b9cfaf927edc1c8" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38009 } } }
I20260807 22:45:41.193331 22255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:41.193533 22264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:41.194128 22264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:41.196142 22264 catalog_manager.cc:1347] Generated new cluster ID: ec5b522c688c45d199badc680b561e1b
I20260807 22:45:41.196213 22264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:45:41.200111 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:41.192945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.16.98.190:38009)
0807 22:45:41.193685 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:41.193696 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:41.193733 (+ 37us) server_negotiation.cc:407] Connection header received
0807 22:45:41.193878 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:41.193886 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:41.193969 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:41.194132 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:41.194507 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.195829 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.197267 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.198007 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.198127 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:41.199197 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:41.199202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:41.199203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:41.199431 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:41.199476 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:41.199491 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:41.199562 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:41.199772 (+ 210us) server_negotiation.cc:299] Negotiation successful
0807 22:45:41.199860 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":89,"threads_started":1}
I20260807 22:45:41.201725 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:45:41.206482 22264 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:41.207263 22264 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:41.215235 22264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44ded3dfdcb143349b9cfaf927edc1c8: Generated new TSK 0
I20260807 22:45:41.215485 22264 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 22:45:41.240343 22282 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260807 22:45:41.240458 22282 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 22:45:41.240885 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:41.224845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.16.98.190:38009)
0807 22:45:41.224899 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:41.224902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:41.225023 (+ 121us) server_negotiation.cc:407] Connection header received
0807 22:45:41.225110 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:41.225111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:41.225124 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:41.225194 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0807 22:45:41.225358 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.225951 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.227231 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.227496 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.227564 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:41.227771 (+ 207us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 22:45:41.240639 (+ 12868us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 22:45:41.240810 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54322: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":28}
W20260807 22:45:41.240989 22263 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54322: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260807 22:45:41.241159 22281 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:41.224743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:38009 (local address 127.0.0.1:54322)
0807 22:45:41.224930 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:41.224950 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:41.225030 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:41.225214 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:41.225215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:41.225217 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0807 22:45:41.225282 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:41.225285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.226043 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.226045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:41.227122 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:41.227126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.227585 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.227587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:41.227653 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:41.227659 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 22:45:41.240797 (+ 13138us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 22:45:41.240956 (+ 159us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.98.190:38009: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39517/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":110,"threads_started":1}
I20260807 22:45:41.241611 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 22182
2026-08-07T22:45:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (269 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260807 22:45:41.282236 16778 mini_oidc.cc:152] Starting JWKS server
I20260807 22:45:41.282971 16778 webserver.cc:460] Webserver started at https://127.0.0.1:40665/ using document root <none> and password file <none>
I20260807 22:45:41.283043 16778 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 22:45:41.283252 16778 webserver.cc:460] Webserver started at http://127.0.0.1:44967/ using document root <none> and password file <none>
2026-08-07T22:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:41Z Disabled control of system clock
I20260807 22:45:41.293038 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:38477
--webserver_interface=127.16.98.190
--webserver_port=0
--builtin_ntp_servers=127.16.98.148:42857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:38477
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40665/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 22:45:41.389573 22288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:41.389825 22288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:41.389851 22288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:41.391049 22288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:41.391093 22288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:41.391106 22288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:41.391119 22288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:41.391134 22288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 22:45:41.391146 22288 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 22:45:41.391158 22288 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40665/jwks.json
I20260807 22:45:41.392544 22288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:38477
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:38477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40665/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:41.392807 22288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:41.393083 22288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:41.399682 22293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:41.399690 22294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:41.399690 22296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:41.400064 22288 server_base.cc:1034] running on GCE node
I20260807 22:45:41.400358 22288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:41.400787 22288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:41.402064 22288 hybrid_clock.cc:648] HybridClock initialized: now 1786142741402036 us; error 43 us; skew 500 ppm
I20260807 22:45:41.403795 22288 webserver.cc:460] Webserver started at http://127.16.98.190:32875/ using document root <none> and password file <none>
I20260807 22:45:41.404142 22288 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:41.404210 22288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:41.404366 22288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:41.405668 22288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "6baf12c8f3c046839edb33f3768465d3"
format_stamp: "Formatted at 2026-08-07 22:45:41 on dist-test-slave-rgb1"
server_key: "d91e24f4dc97a3bd3dce5b705905bcb1"
server_key_iv: "96ff9327ca61005b53aa41f4c38702c2"
server_key_version: "encryptionkey@0"
I20260807 22:45:41.406144 22288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "6baf12c8f3c046839edb33f3768465d3"
format_stamp: "Formatted at 2026-08-07 22:45:41 on dist-test-slave-rgb1"
server_key: "d91e24f4dc97a3bd3dce5b705905bcb1"
server_key_iv: "96ff9327ca61005b53aa41f4c38702c2"
server_key_version: "encryptionkey@0"
I20260807 22:45:41.408228 22288 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 22:45:41.409512 22302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:41.409773 22288 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:41.409881 22288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "6baf12c8f3c046839edb33f3768465d3"
format_stamp: "Formatted at 2026-08-07 22:45:41 on dist-test-slave-rgb1"
server_key: "d91e24f4dc97a3bd3dce5b705905bcb1"
server_key_iv: "96ff9327ca61005b53aa41f4c38702c2"
server_key_version: "encryptionkey@0"
I20260807 22:45:41.409946 22288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:41.424328 22288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:41.424798 22288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:41.424937 22288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:41.430166 22288 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:38477
I20260807 22:45:41.430210 22356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:38477 every 8 connection(s)
I20260807 22:45:41.430613 22288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:41.431538 22357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:41.434835 22357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3: Bootstrap starting.
I20260807 22:45:41.435706 22357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:41.436094 22357 log.cc:824] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:41.437319 22357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3: No bootstrap required, opened a new log
I20260807 22:45:41.439142 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 22288
I20260807 22:45:41.439291 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:41.439487 16778 external_mini_cluster.cc:1428] Setting key f3340edef6bd899717e4715a732f969b
I20260807 22:45:41.439620 22357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6baf12c8f3c046839edb33f3768465d3" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38477 } }
I20260807 22:45:41.439816 22357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:41.439844 22357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6baf12c8f3c046839edb33f3768465d3, State: Initialized, Role: FOLLOWER
I20260807 22:45:41.440014 22357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6baf12c8f3c046839edb33f3768465d3" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38477 } }
I20260807 22:45:41.440114 22357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:41.440165 22357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:41.440231 22357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:41.441179 22357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6baf12c8f3c046839edb33f3768465d3" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38477 } }
I20260807 22:45:41.441288 22357 leader_election.cc:302] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6baf12c8f3c046839edb33f3768465d3; no voters:
I20260807 22:45:41.441532 22357 leader_election.cc:288] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:41.441797 22362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:41.441860 22357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:41.441962 22362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [term 1 LEADER]: Becoming Leader. State: Replica: 6baf12c8f3c046839edb33f3768465d3, State: Running, Role: LEADER
I20260807 22:45:41.442054 22362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6baf12c8f3c046839edb33f3768465d3" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38477 } }
I20260807 22:45:41.442385 22363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6baf12c8f3c046839edb33f3768465d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6baf12c8f3c046839edb33f3768465d3" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38477 } } }
I20260807 22:45:41.442634 22363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:41.442394 22364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6baf12c8f3c046839edb33f3768465d3. Latest consensus state: current_term: 1 leader_uuid: "6baf12c8f3c046839edb33f3768465d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6baf12c8f3c046839edb33f3768465d3" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38477 } } }
I20260807 22:45:41.443099 22364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:41.443252 22371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:41.443774 22371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:41.445745 22371 catalog_manager.cc:1347] Generated new cluster ID: dbc5c952b07e49c280340fdc61acc96a
I20260807 22:45:41.445807 22371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:45:41.446564 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:41.440104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.16.98.190:38477)
0807 22:45:41.440450 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:41.440459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:41.440485 (+ 26us) server_negotiation.cc:407] Connection header received
0807 22:45:41.440659 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:41.440665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:41.440727 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:41.440874 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:41.441129 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.442626 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.443963 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.444793 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.444928 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:41.445758 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:41.445762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:41.445762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:41.445999 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:41.446043 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:41.446059 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:41.446141 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:41.446239 (+ 98us) server_negotiation.cc:299] Negotiation successful
0807 22:45:41.446332 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":132,"threads_started":1}
I20260807 22:45:41.447878 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:45:41.484289 22371 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:41.485173 22371 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:41.489895 22371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6baf12c8f3c046839edb33f3768465d3: Generated new TSK 0
I20260807 22:45:41.490229 22371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:45:41.510447 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:41.506231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.16.98.190:38477)
0807 22:45:41.506300 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:41.506302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:41.506472 (+ 170us) server_negotiation.cc:407] Connection header received
0807 22:45:41.506570 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:41.506571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:41.506585 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:41.506671 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0807 22:45:41.506843 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.507569 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.508426 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.508787 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.508878 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:41.509111 (+ 233us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 22:45:41.510185 (+ 1074us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 22:45:41.510362 (+ 177us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53260: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":40}
W20260807 22:45:41.510574 22386 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:41.506131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:38477 (local address 127.0.0.1:53260)
0807 22:45:41.506382 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:41.506403 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:41.506489 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:41.506707 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:41.506708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:41.506710 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0807 22:45:41.506766 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:41.506768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.507684 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.507685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:41.508324 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:41.508327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.508912 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.508913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:41.508970 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:41.508978 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 22:45:41.510315 (+ 1337us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 22:45:41.510424 (+ 109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.98.190:38477: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":155,"threads_started":1}
W20260807 22:45:41.510915 22361 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53260: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260807 22:45:41.512425 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 22288
E20260807 22:45:41.521555 22387 webserver.cc:533] Webserver: sslize failed:
E20260807 22:45:41.521651 22387 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-07T22:45:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (269 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-07T22:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:41Z Disabled control of system clock
I20260807 22:45:41.563220 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:38869
--webserver_interface=127.16.98.190
--webserver_port=0
--builtin_ntp_servers=127.16.98.148:33221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:38869
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/testchainca.cert with env {}
W20260807 22:45:41.652868 22391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:41.653149 22391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:41.653193 22391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:41.654464 22391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:41.654544 22391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:41.654568 22391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:41.654601 22391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:41.654651 22391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 22:45:41.654680 22391 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 22:45:41.654706 22391 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260807 22:45:41.656358 22391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:38869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:38869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:41.656693 22391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:41.656941 22391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:41.662797 22397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:41.662792 22400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:41.662959 22398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:41.663146 22391 server_base.cc:1034] running on GCE node
I20260807 22:45:41.663376 22391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:41.663745 22391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:41.664945 22391 hybrid_clock.cc:648] HybridClock initialized: now 1786142741664917 us; error 42 us; skew 500 ppm
I20260807 22:45:41.666574 22391 webserver.cc:460] Webserver started at http://127.16.98.190:46369/ using document root <none> and password file <none>
I20260807 22:45:41.666887 22391 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:41.666966 22391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:41.667130 22391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:41.668318 22391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "df9e190dd8e04c7fb7768a04e6885ca9"
format_stamp: "Formatted at 2026-08-07 22:45:41 on dist-test-slave-rgb1"
server_key: "50827fdb885ab91e695e9aad6fedb7f2"
server_key_iv: "384c7cc4a7652a09feb3faaf23e5d679"
server_key_version: "encryptionkey@0"
I20260807 22:45:41.668752 22391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "df9e190dd8e04c7fb7768a04e6885ca9"
format_stamp: "Formatted at 2026-08-07 22:45:41 on dist-test-slave-rgb1"
server_key: "50827fdb885ab91e695e9aad6fedb7f2"
server_key_iv: "384c7cc4a7652a09feb3faaf23e5d679"
server_key_version: "encryptionkey@0"
I20260807 22:45:41.670404 22391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:41.671437 22407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:41.671680 22391 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:41.671789 22391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "df9e190dd8e04c7fb7768a04e6885ca9"
format_stamp: "Formatted at 2026-08-07 22:45:41 on dist-test-slave-rgb1"
server_key: "50827fdb885ab91e695e9aad6fedb7f2"
server_key_iv: "384c7cc4a7652a09feb3faaf23e5d679"
server_key_version: "encryptionkey@0"
I20260807 22:45:41.671859 22391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:41.686329 22391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:41.686655 22391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:41.686769 22391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:41.691666 22391 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:38869
I20260807 22:45:41.691744 22460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:38869 every 8 connection(s)
I20260807 22:45:41.692138 22391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:41.693325 22461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:41.695797 22461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9: Bootstrap starting.
I20260807 22:45:41.696458 22461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:41.696844 22461 log.cc:824] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:41.697916 22461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9: No bootstrap required, opened a new log
I20260807 22:45:41.699246 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 22391
I20260807 22:45:41.699348 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:41.699589 16778 external_mini_cluster.cc:1428] Setting key 7aa855f1a27093344374b08745c79dd8
I20260807 22:45:41.700217 22461 raft_consensus.cc:348] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9e190dd8e04c7fb7768a04e6885ca9" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38869 } }
I20260807 22:45:41.700419 22461 raft_consensus.cc:374] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:41.700481 22461 raft_consensus.cc:729] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df9e190dd8e04c7fb7768a04e6885ca9, State: Initialized, Role: FOLLOWER
I20260807 22:45:41.700682 22461 consensus_queue.cc:260] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9e190dd8e04c7fb7768a04e6885ca9" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38869 } }
I20260807 22:45:41.700788 22461 raft_consensus.cc:388] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:41.700834 22461 raft_consensus.cc:482] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:41.700903 22461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:41.701571 22461 raft_consensus.cc:504] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9e190dd8e04c7fb7768a04e6885ca9" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38869 } }
I20260807 22:45:41.701707 22461 leader_election.cc:302] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df9e190dd8e04c7fb7768a04e6885ca9; no voters:
I20260807 22:45:41.701934 22461 leader_election.cc:288] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:41.701972 22466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:41.702107 22466 raft_consensus.cc:686] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [term 1 LEADER]: Becoming Leader. State: Replica: df9e190dd8e04c7fb7768a04e6885ca9, State: Running, Role: LEADER
I20260807 22:45:41.702199 22466 consensus_queue.cc:237] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9e190dd8e04c7fb7768a04e6885ca9" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38869 } }
I20260807 22:45:41.702244 22461 sys_catalog.cc:564] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:41.702582 22466 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df9e190dd8e04c7fb7768a04e6885ca9. Latest consensus state: current_term: 1 leader_uuid: "df9e190dd8e04c7fb7768a04e6885ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9e190dd8e04c7fb7768a04e6885ca9" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38869 } } }
I20260807 22:45:41.702675 22466 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:41.702594 22467 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df9e190dd8e04c7fb7768a04e6885ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9e190dd8e04c7fb7768a04e6885ca9" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 38869 } } }
I20260807 22:45:41.702842 22467 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:41.702921 22471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:41.703588 22471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:41.706249 22471 catalog_manager.cc:1347] Generated new cluster ID: ea055e9da85d4ffb829ee762212a14b2
I20260807 22:45:41.706313 22471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:45:41.706373 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:41.700270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47966 (local address 127.16.98.190:38869)
0807 22:45:41.700594 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:41.700605 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:41.700639 (+ 34us) server_negotiation.cc:407] Connection header received
0807 22:45:41.700745 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:41.700751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:41.700836 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:41.700983 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:41.701219 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.702836 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.703915 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.704535 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.704698 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:41.705474 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:41.705478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:41.705479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:45:41.705754 (+ 275us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:41.705802 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:45:41.705817 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:41.705926 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:41.706011 (+ 85us) server_negotiation.cc:299] Negotiation successful
0807 22:45:41.706103 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":134,"threads_started":1}
I20260807 22:45:41.707507 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:45:41.736074 22471 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:41.736789 22471 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:41.741577 22471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df9e190dd8e04c7fb7768a04e6885ca9: Generated new TSK 0
I20260807 22:45:41.741809 22471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:45:42.193460 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:41.756821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.16.98.190:38869)
0807 22:45:41.756873 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:41.756875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:41.756962 (+ 87us) server_negotiation.cc:407] Connection header received
0807 22:45:41.757102 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:41.757103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:41.757116 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:41.757187 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0807 22:45:41.757343 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.757824 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.759121 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.759327 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.759386 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:41.759591 (+ 205us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 22:45:42.193181 (+433590us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 22:45:42.193395 (+ 214us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47978: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":27}
W20260807 22:45:42.193568 22465 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47978: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260807 22:45:42.193765 22491 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:41.756707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:38869 (local address 127.0.0.1:47978)
0807 22:45:41.756880 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:41.756893 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:41.756967 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:41.757204 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:41.757205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:41.757207 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0807 22:45:41.757273 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:41.757276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.757922 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.757924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:41.758999 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:41.759003 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:41.759414 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:41.759415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:41.759483 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:41.759489 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 22:45:42.193414 (+433925us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 22:45:42.193589 (+ 175us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.98.190:38869: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":106,"threads_started":1}
I20260807 22:45:42.194918 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 22391
2026-08-07T22:45:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (663 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (106 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (113 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:45:43 dist-test-slave-rgb1 krb5kdc[22503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:45:43 dist-test-slave-rgb1 krb5kdc[22503](info): set up 2 sockets
Aug 07 22:45:43 dist-test-slave-rgb1 krb5kdc[22503](info): commencing operation
krb5kdc: starting...
W20260807 22:45:44.897436 16778 mini_kdc.cc:121] Time spent starting KDC: real 2.460s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:45:44 dist-test-slave-rgb1 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:45:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:45:45.053507 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:36233
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:36233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:45.147248 22516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:45.147500 22516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:45.147527 22516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:45.148774 22516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:45.148821 22516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:45.148836 22516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:45.148849 22516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:45.148862 22516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:45.150380 22516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:36233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:36233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:45.150619 22516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:45.150813 22516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:45.157176 22521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:45.157352 22516 server_base.cc:1034] running on GCE node
W20260807 22:45:45.157404 22524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:45.157187 22522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:45.157867 22516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:45.158195 22516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:45.159358 22516 hybrid_clock.cc:648] HybridClock initialized: now 1786142745159317 us; error 61 us; skew 500 ppm
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:45.167557 22516 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:45:45.168031 22516 webserver.cc:460] Webserver started at http://127.16.98.190:40549/ using document root <none> and password file <none>
I20260807 22:45:45.168228 22516 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:45.168272 22516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:45.168370 22516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:45.169864 22516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "dd1cb41a18ab43ad973bb23f8c1fdaf0"
format_stamp: "Formatted at 2026-08-07 22:45:45 on dist-test-slave-rgb1"
server_key: "c581859d2acac1d6a8d5e7b17f80268f"
server_key_iv: "e099ff3510ff1916a98a667c60485e87"
server_key_version: "encryptionkey@0"
I20260807 22:45:45.170225 22516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "dd1cb41a18ab43ad973bb23f8c1fdaf0"
format_stamp: "Formatted at 2026-08-07 22:45:45 on dist-test-slave-rgb1"
server_key: "c581859d2acac1d6a8d5e7b17f80268f"
server_key_iv: "e099ff3510ff1916a98a667c60485e87"
server_key_version: "encryptionkey@0"
I20260807 22:45:45.171928 22516 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260807 22:45:45.173167 22531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:45.173401 22516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:45.173511 22516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "dd1cb41a18ab43ad973bb23f8c1fdaf0"
format_stamp: "Formatted at 2026-08-07 22:45:45 on dist-test-slave-rgb1"
server_key: "c581859d2acac1d6a8d5e7b17f80268f"
server_key_iv: "e099ff3510ff1916a98a667c60485e87"
server_key_version: "encryptionkey@0"
I20260807 22:45:45.173581 22516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:45.183921 22516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:45.186563 22516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:45.186707 22516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:45.191491 22516 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:36233
I20260807 22:45:45.191568 22585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:36233 every 8 connection(s)
I20260807 22:45:45.191962 22516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:45.193111 22586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:45.196123 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0: Bootstrap starting.
I20260807 22:45:45.197183 22586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:45.197598 22586 log.cc:824] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:45.199046 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0: No bootstrap required, opened a new log
I20260807 22:45:45.200193 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 22516
I20260807 22:45:45.200346 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:45.200763 16778 external_mini_cluster.cc:1428] Setting key efabafb700e0ebfc82ffcd9b55aa0ca5
I20260807 22:45:45.201943 22586 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1cb41a18ab43ad973bb23f8c1fdaf0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36233 } }
I20260807 22:45:45.202193 22586 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:45.202242 22586 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1cb41a18ab43ad973bb23f8c1fdaf0, State: Initialized, Role: FOLLOWER
I20260807 22:45:45.202401 22586 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1cb41a18ab43ad973bb23f8c1fdaf0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36233 } }
I20260807 22:45:45.202512 22586 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:45.202584 22586 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:45.202644 22586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:45.204015 22586 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1cb41a18ab43ad973bb23f8c1fdaf0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36233 } }
I20260807 22:45:45.204144 22586 leader_election.cc:302] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd1cb41a18ab43ad973bb23f8c1fdaf0; no voters:
I20260807 22:45:45.204347 22586 leader_election.cc:288] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:45.204522 22591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:45.204749 22586 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:45.204967 22591 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [term 1 LEADER]: Becoming Leader. State: Replica: dd1cb41a18ab43ad973bb23f8c1fdaf0, State: Running, Role: LEADER
I20260807 22:45:45.205219 22591 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1cb41a18ab43ad973bb23f8c1fdaf0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36233 } }
I20260807 22:45:45.205694 22592 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd1cb41a18ab43ad973bb23f8c1fdaf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1cb41a18ab43ad973bb23f8c1fdaf0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36233 } } }
I20260807 22:45:45.205791 22592 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:45.206166 22593 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd1cb41a18ab43ad973bb23f8c1fdaf0. Latest consensus state: current_term: 1 leader_uuid: "dd1cb41a18ab43ad973bb23f8c1fdaf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1cb41a18ab43ad973bb23f8c1fdaf0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36233 } } }
I20260807 22:45:45.206243 22593 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:45.206657 22600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:45.207784 22600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:45.209812 22600 catalog_manager.cc:1347] Generated new cluster ID: 5b89e8536a09415a8b4afb95b55c6d13
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:45.209874 22600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:45:45.215324 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:45.201582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.16.98.190:36233)
0807 22:45:45.201863 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:45.201874 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:45.201914 (+ 40us) server_negotiation.cc:407] Connection header received
0807 22:45:45.202045 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:45.202052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:45.202121 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:45.202287 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:45.203210 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.204692 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.205918 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.206718 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.207182 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:45.210514 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:45.210518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:45.210519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:45.210532 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:45.212978 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:45.213480 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:45.213483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:45.213484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:45.213645 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:45.213881 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:45.213882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:45.213882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:45.214224 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:45.214897 (+ 673us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:45.214968 (+ 71us) server_negotiation.cc:299] Negotiation successful
0807 22:45:45.215109 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":140,"threads_started":1}
I20260807 22:45:45.231444 22600 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:45.232182 22600 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:45.239614 22600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd1cb41a18ab43ad973bb23f8c1fdaf0: Generated new TSK 0
I20260807 22:45:45.239845 22600 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:45:45.273639 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36233
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:45.366606 22612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:45.366879 22612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:45.366910 22612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:45.368346 22612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:45.368412 22612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:45.368446 22612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:45:45.371038 22612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:45.371362 22612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:45.372131 22612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:45.378677 22620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:45.378656 22617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:45.378728 22618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:45.379065 22612 server_base.cc:1034] running on GCE node
I20260807 22:45:45.379299 22612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:45.379595 22612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:45.380820 22612 hybrid_clock.cc:648] HybridClock initialized: now 1786142745380803 us; error 43 us; skew 500 ppm
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:45.383941 22612 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:45:45.384656 22612 webserver.cc:460] Webserver started at http://127.16.98.129:45109/ using document root <none> and password file <none>
I20260807 22:45:45.385002 22612 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:45.385092 22612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:45.385267 22612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:45.386483 22612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "dfad6dbddc174f24bf49d29bb7569584"
format_stamp: "Formatted at 2026-08-07 22:45:45 on dist-test-slave-rgb1"
server_key: "33bd5df145a1c27245ffe1d983930941"
server_key_iv: "6ddaee13e15b772ecc99a886a0f29dc9"
server_key_version: "encryptionkey@0"
I20260807 22:45:45.386926 22612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "dfad6dbddc174f24bf49d29bb7569584"
format_stamp: "Formatted at 2026-08-07 22:45:45 on dist-test-slave-rgb1"
server_key: "33bd5df145a1c27245ffe1d983930941"
server_key_iv: "6ddaee13e15b772ecc99a886a0f29dc9"
server_key_version: "encryptionkey@0"
I20260807 22:45:45.388618 22612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260807 22:45:45.389590 22627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:45.389801 22612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:45.389981 22612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "dfad6dbddc174f24bf49d29bb7569584"
format_stamp: "Formatted at 2026-08-07 22:45:45 on dist-test-slave-rgb1"
server_key: "33bd5df145a1c27245ffe1d983930941"
server_key_iv: "6ddaee13e15b772ecc99a886a0f29dc9"
server_key_version: "encryptionkey@0"
I20260807 22:45:45.390076 22612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:45.409482 22612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:45.412257 22612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:45.412428 22612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:45.412766 22612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:45.413166 22612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:45.413232 22612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:45.413285 22612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:45.413311 22612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:45.421870 22612 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:42927
I20260807 22:45:45.421950 22743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:42927 every 8 connection(s)
I20260807 22:45:45.422346 22612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:45.431159 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 22612
I20260807 22:45:45.431327 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:45:45.431617 16778 external_mini_cluster.cc:1428] Setting key 199777db6f8be8586fd5cbf3a9b9236b
I20260807 22:45:45.434248 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:45.423253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:36871 (local address 127.16.98.190:36233)
0807 22:45:45.423349 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:45.423352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:45.423454 (+ 102us) server_negotiation.cc:407] Connection header received
0807 22:45:45.424458 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:45.424459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:45.424473 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:45.424536 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:45.425366 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.426020 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.427160 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.427415 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.427501 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:45.430024 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:45.430026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:45.430026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:45.430035 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:45.431938 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:45.432714 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:45.432715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:45.432716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:45.432807 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:45.433205 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:45.433206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:45.433207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:45.433364 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:45.433449 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:45.433856 (+ 407us) server_negotiation.cc:299] Negotiation successful
0807 22:45:45.433956 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 22:45:45.434298 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:45.423153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36233 (local address 127.16.98.129:36871)
0807 22:45:45.423317 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:45.423371 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:45.424322 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:45.424671 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:45.424676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:45.424967 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:45.425241 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:45.425246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.426139 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.426140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:45.427050 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:45.427053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.427533 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.427534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:45.427654 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:45.428292 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:45.428301 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:45.429853 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:45.432129 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:45.432140 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:45.432143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:45.432421 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:45.432889 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:45.432890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:45.432891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:45.433074 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:45.433456 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:45.433464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:45.433667 (+ 203us) client_negotiation.cc:769] Sending connection context
0807 22:45:45.433797 (+ 130us) client_negotiation.cc:240] Negotiation successful
0807 22:45:45.434002 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260807 22:45:45.435128 22744 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36233
I20260807 22:45:45.435231 22744 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:45.435451 22744 heartbeater.cc:511] Master 127.16.98.190:36233 requested a full tablet report, sending...
I20260807 22:45:45.436079 22549 ts_manager.cc:194] Registered new tserver with Master: dfad6dbddc174f24bf49d29bb7569584 (127.16.98.129:42927)
I20260807 22:45:45.437063 22549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:36871
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:45:45.465675 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36233
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:45.550547 22749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:45.550819 22749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:45.550870 22749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:45.552078 22749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:45.552155 22749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:45.552202 22749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:45:45.553861 22749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:45.554162 22749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:45.554410 22749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:45.559804 22756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:45.559816 22758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:45.559816 22755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:45.560217 22749 server_base.cc:1034] running on GCE node
I20260807 22:45:45.560492 22749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:45.560783 22749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:45.561966 22749 hybrid_clock.cc:648] HybridClock initialized: now 1786142745561982 us; error 69 us; skew 500 ppm
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:45.565012 22749 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:45:45.565478 22749 webserver.cc:460] Webserver started at http://127.16.98.130:42323/ using document root <none> and password file <none>
I20260807 22:45:45.565697 22749 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:45.565745 22749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:45.565846 22749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:45.566963 22749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "d193e728c9774bb1b2409c49867fe088"
format_stamp: "Formatted at 2026-08-07 22:45:45 on dist-test-slave-rgb1"
server_key: "b55aa0eb334e4d2ae166e3d1b068f813"
server_key_iv: "91881cd6da28ef9ec6f11cc74e7f293f"
server_key_version: "encryptionkey@0"
I20260807 22:45:45.567328 22749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "d193e728c9774bb1b2409c49867fe088"
format_stamp: "Formatted at 2026-08-07 22:45:45 on dist-test-slave-rgb1"
server_key: "b55aa0eb334e4d2ae166e3d1b068f813"
server_key_iv: "91881cd6da28ef9ec6f11cc74e7f293f"
server_key_version: "encryptionkey@0"
I20260807 22:45:45.568868 22749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260807 22:45:45.569831 22765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:45.570266 22749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:45.570408 22749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "d193e728c9774bb1b2409c49867fe088"
format_stamp: "Formatted at 2026-08-07 22:45:45 on dist-test-slave-rgb1"
server_key: "b55aa0eb334e4d2ae166e3d1b068f813"
server_key_iv: "91881cd6da28ef9ec6f11cc74e7f293f"
server_key_version: "encryptionkey@0"
I20260807 22:45:45.570474 22749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:45.588903 22749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:45.591625 22749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:45.591763 22749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:45.592092 22749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:45.592458 22749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:45.592497 22749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:45.592522 22749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:45.592628 22749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:45.599519 22749 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:46009
I20260807 22:45:45.599567 22881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:46009 every 8 connection(s)
I20260807 22:45:45.599917 22749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:45:45.602085 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 22749
I20260807 22:45:45.602246 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:45:45.602478 16778 external_mini_cluster.cc:1428] Setting key 9f708ac119646700cb4cc9fb9a42d239
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:45.611603 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:45.600871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:54667 (local address 127.16.98.190:36233)
0807 22:45:45.600946 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:45.600948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:45.601170 (+ 222us) server_negotiation.cc:407] Connection header received
0807 22:45:45.602162 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:45.602164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:45.602179 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:45.602285 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:45.603098 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.603659 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.604834 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.605039 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.605127 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:45.607931 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:45.607932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:45.607932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:45.607940 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:45.609649 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:45.610237 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:45.610238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:45.610238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:45.610300 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:45.610660 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:45.610662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:45.610662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:45.610802 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:45.610962 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:45.611407 (+ 445us) server_negotiation.cc:299] Negotiation successful
0807 22:45:45.611522 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 22:45:45.611824 22885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:45.600851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36233 (local address 127.16.98.130:54667)
0807 22:45:45.601041 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:45.601079 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:45.602045 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:45.602379 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:45.602384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:45.602778 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:45.603001 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:45.603005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.603766 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.603767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:45.604735 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:45.604738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.605139 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.605141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:45.605278 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:45.605942 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:45.605950 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:45.607765 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:45.609749 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:45.609760 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:45.609763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:45.610123 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:45.610387 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:45.610388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:45.610388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:45.610549 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:45.610995 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:45.611004 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:45.611210 (+ 206us) client_negotiation.cc:769] Sending connection context
0807 22:45:45.611347 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 22:45:45.611547 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":105,"threads_started":1}
I20260807 22:45:45.612416 22882 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36233
I20260807 22:45:45.612557 22882 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:45.612813 22882 heartbeater.cc:511] Master 127.16.98.190:36233 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
I20260807 22:45:45.613315 22549 ts_manager.cc:194] Registered new tserver with Master: d193e728c9774bb1b2409c49867fe088 (127.16.98.130:46009)
I20260807 22:45:45.613760 22549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:54667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:45:45.636757 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36233
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:45.725642 22888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:45.725939 22888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:45.725984 22888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:45.727339 22888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:45.727415 22888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:45.727466 22888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:45:45.729141 22888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:45.729466 22888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:45.729734 22888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:45.735204 22893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:45.735174 22894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:45.735174 22896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:45.735267 22888 server_base.cc:1034] running on GCE node
I20260807 22:45:45.735757 22888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:45.736059 22888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:45.737308 22888 hybrid_clock.cc:648] HybridClock initialized: now 1786142745737243 us; error 90 us; skew 500 ppm
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:45.740504 22888 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:45:45.740999 22888 webserver.cc:460] Webserver started at http://127.16.98.131:41817/ using document root <none> and password file <none>
I20260807 22:45:45.741225 22888 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:45.741269 22888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:45.741361 22888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:45.742396 22888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "a93a18a5f2f5444ead4b9ab9a379009a"
format_stamp: "Formatted at 2026-08-07 22:45:45 on dist-test-slave-rgb1"
server_key: "7af303a46dcb24ffbb9a1b3590b94aa8"
server_key_iv: "d1804782c69c00400075eaf22797db3f"
server_key_version: "encryptionkey@0"
I20260807 22:45:45.742753 22888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "a93a18a5f2f5444ead4b9ab9a379009a"
format_stamp: "Formatted at 2026-08-07 22:45:45 on dist-test-slave-rgb1"
server_key: "7af303a46dcb24ffbb9a1b3590b94aa8"
server_key_iv: "d1804782c69c00400075eaf22797db3f"
server_key_version: "encryptionkey@0"
I20260807 22:45:45.744277 22888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260807 22:45:45.745250 22904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:45.745549 22888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:45.745640 22888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "a93a18a5f2f5444ead4b9ab9a379009a"
format_stamp: "Formatted at 2026-08-07 22:45:45 on dist-test-slave-rgb1"
server_key: "7af303a46dcb24ffbb9a1b3590b94aa8"
server_key_iv: "d1804782c69c00400075eaf22797db3f"
server_key_version: "encryptionkey@0"
I20260807 22:45:45.745744 22888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:45.764366 22888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:45.766788 22888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:45.766927 22888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:45.767189 22888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:45.767544 22888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:45.767606 22888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:45.767650 22888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:45.767680 22888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:45.775172 22888 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:40371
I20260807 22:45:45.775285 23022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:40371 every 8 connection(s)
I20260807 22:45:45.775612 22888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:45.783711 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 22888
I20260807 22:45:45.783861 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:45:45.784152 16778 external_mini_cluster.cc:1428] Setting key 50d9298e47e10ed591b0311fba936082
I20260807 22:45:45.786823 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:45.776659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:57681 (local address 127.16.98.190:36233)
0807 22:45:45.776733 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:45.776736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:45.776905 (+ 169us) server_negotiation.cc:407] Connection header received
0807 22:45:45.778063 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:45.778064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:45.778077 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:45.778159 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:45.778836 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.779345 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.780374 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.780654 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.780748 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:45.783253 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:45.783254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:45.783254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:45.783262 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:45.785378 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:45.785846 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:45.785847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:45.785847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:45.785907 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:45.786203 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:45.786204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:45.786204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:45.786325 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:45.786395 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:45.786674 (+ 279us) server_negotiation.cc:299] Negotiation successful
0807 22:45:45.786752 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 22:45:45.787106 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:45.776519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36233 (local address 127.16.98.131:57681)
0807 22:45:45.776764 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:45.776811 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:45.777892 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:45.778236 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:45.778240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:45.778538 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:45.778731 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:45.778735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.779452 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.779453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:45.780265 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:45.780267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.780761 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.780762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:45.780893 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:45.781416 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:45.781424 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:45.783136 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:45.785479 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:45.785487 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:45.785489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:45.785757 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:45.786000 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:45.786001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:45.786001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:45.786126 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:45.786407 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:45.786413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:45.786551 (+ 138us) client_negotiation.cc:769] Sending connection context
0807 22:45:45.786642 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 22:45:45.786808 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":105,"threads_started":1}
I20260807 22:45:45.787678 23023 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36233
I20260807 22:45:45.787815 23023 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:45.788091 23023 heartbeater.cc:511] Master 127.16.98.190:36233 requested a full tablet report, sending...
I20260807 22:45:45.788605 22549 ts_manager.cc:194] Registered new tserver with Master: a93a18a5f2f5444ead4b9ab9a379009a (127.16.98.131:40371)
I20260807 22:45:45.789002 22549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:57681
I20260807 22:45:45.796550 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/test-admin.keytab.
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:45.818640 16778 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260807 22:45:45.821133 23031 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:45.819830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.16.98.129:42927)
0807 22:45:45.820135 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:45.820139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:45.820151 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:45:45.820178 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:45.820179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:45.820194 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:45.820283 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:45.821010 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53682: BlockingRecv error: recv got EOF from 127.0.0.1:53682 (error 108)
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":199,"threads_started":1}
W20260807 22:45:45.821141 16778 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/krb5cc' of type FILE
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:45.822386 16778 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:45.823535 16778 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.129@KRBTEST.COM
I20260807 22:45:45.833099 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:45.824273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53692 (local address 127.16.98.129:42927)
0807 22:45:45.824336 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:45.824338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:45.824347 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:45:45.824378 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:45.824379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:45.824398 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:45.824467 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:45.825000 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.826061 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.826850 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.827209 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.827300 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:45.829650 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:45.829661 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:45.829661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:45.829676 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:45.831463 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:45.832029 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:45.832033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:45.832033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:45.832108 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:45.832336 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:45.832338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:45.832338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:45.832511 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:45.832782 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:45.832872 (+ 90us) server_negotiation.cc:299] Negotiation successful
0807 22:45:45.832968 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260807 22:45:45.836045 16778 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/krb5cc' of type FILE
W20260807 22:45:45.836053 23031 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:45.835010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53698 (local address 127.16.98.129:42927)
0807 22:45:45.835099 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:45.835101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:45.835111 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:45:45.835170 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:45.835170 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:45.835182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:45.835225 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:45.835996 (+ 771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53698: BlockingRecv error: recv got EOF from 127.0.0.1:53698 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:45.837430 16778 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:45.838534 16778 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.129@KRBTEST.COM
I20260807 22:45:45.848110 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:45.839193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53712 (local address 127.16.98.129:42927)
0807 22:45:45.839291 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:45.839294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:45.839387 (+ 93us) server_negotiation.cc:407] Connection header received
0807 22:45:45.839496 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:45.839500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:45.839518 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:45.839593 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:45.840119 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.840840 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.841674 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.841918 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.842010 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:45.844452 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:45.844453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:45.844454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:45.844461 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:45.846423 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:45.847238 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:45.847240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:45.847240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:45.847306 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:45.847547 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:45.847548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:45.847549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:45.847682 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:45.847773 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:45.847944 (+ 171us) server_negotiation.cc:299] Negotiation successful
0807 22:45:45.848039 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260807 22:45:45.850131 23031 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:45.849428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.16.98.129:42927)
0807 22:45:45.849521 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:45.849523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:45.849604 (+ 81us) server_negotiation.cc:407] Connection header received
0807 22:45:45.849668 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:45.849669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:45.849678 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:45.849727 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:45.850072 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53722: BlockingRecv error: recv got EOF from 127.0.0.1:53722 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260807 22:45:45.850286 16778 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786142716408518-16778-0/krb5kdc/krb5cc' of type FILE
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:45.851624 16778 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:45.853070 16778 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[22503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.129@KRBTEST.COM
I20260807 22:45:45.863591 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:45.853757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.16.98.129:42927)
0807 22:45:45.853882 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:45.853884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:45.853896 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:45:45.853976 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:45.853976 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:45.853991 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:45.854054 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:45.854641 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.855224 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.855998 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:45.856240 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:45.856316 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:45.859244 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:45.859245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:45.859245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:45.859253 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:45.861526 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:45.862212 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:45.862214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:45.862218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:45.862332 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:45.862653 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:45.862654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:45.862654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:45.862787 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:45.863310 (+ 523us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:45.863361 (+ 51us) server_negotiation.cc:299] Negotiation successful
0807 22:45:45.863460 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260807 22:45:45.864727 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 22612
I20260807 22:45:45.872915 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 22749
I20260807 22:45:45.881278 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 22888
I20260807 22:45:45.887880 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 22516
2026-08-07T22:45:45Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3474 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[23045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[23045](info): set up 2 sockets
Aug 07 22:45:45 dist-test-slave-rgb1 krb5kdc[23045](info): commencing operation
krb5kdc: starting...
W20260807 22:45:47.060351 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:45:47 dist-test-slave-rgb1 krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:45:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.190.keytab.
Entry for principal oryx/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.190.keytab.
I20260807 22:45:47.229802 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:35275
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.190.keytab
--principal=oryx/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:35275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:47.333294 23058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:47.333616 23058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:47.333669 23058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:47.335084 23058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:47.335167 23058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:47.335196 23058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:47.335222 23058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:47.335254 23058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:47.337071 23058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:35275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.190.keytab
--principal=oryx/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:35275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:47.337406 23058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:47.337692 23058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:47.343616 23063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:47.343618 23064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:47.343840 23058 server_base.cc:1034] running on GCE node
W20260807 22:45:47.343618 23066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:47.344215 23058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:47.344548 23058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:47.345883 23058 hybrid_clock.cc:648] HybridClock initialized: now 1786142747345864 us; error 47 us; skew 500 ppm
Aug 07 22:45:47 dist-test-slave-rgb1 krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142747, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:47.348773 23058 init.cc:377] Logged in from keytab as oryx/127.16.98.190@KRBTEST.COM (short username oryx)
I20260807 22:45:47.349287 23058 webserver.cc:460] Webserver started at http://127.16.98.190:41187/ using document root <none> and password file <none>
I20260807 22:45:47.349531 23058 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:47.349581 23058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:47.349721 23058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:47.350898 23058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "059a0a6fe8314dc4a77f3e93df688221"
format_stamp: "Formatted at 2026-08-07 22:45:47 on dist-test-slave-rgb1"
server_key: "8e756888be620f6b848897d09feea226"
server_key_iv: "b8a33bd385893144f9678f3418aac5f5"
server_key_version: "encryptionkey@0"
I20260807 22:45:47.351281 23058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "059a0a6fe8314dc4a77f3e93df688221"
format_stamp: "Formatted at 2026-08-07 22:45:47 on dist-test-slave-rgb1"
server_key: "8e756888be620f6b848897d09feea226"
server_key_iv: "b8a33bd385893144f9678f3418aac5f5"
server_key_version: "encryptionkey@0"
I20260807 22:45:47.352782 23058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:45:47.353729 23073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:47.354130 23058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:47.354242 23058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "059a0a6fe8314dc4a77f3e93df688221"
format_stamp: "Formatted at 2026-08-07 22:45:47 on dist-test-slave-rgb1"
server_key: "8e756888be620f6b848897d09feea226"
server_key_iv: "b8a33bd385893144f9678f3418aac5f5"
server_key_version: "encryptionkey@0"
I20260807 22:45:47.354318 23058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:47.385087 23058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:47.388332 23058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:47.388602 23058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:47.393273 23058 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:35275
I20260807 22:45:47.393316 23126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:35275 every 8 connection(s)
I20260807 22:45:47.393707 23058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:47.394740 23127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:47.397511 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 23058
I20260807 22:45:47.397651 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:47.397881 16778 external_mini_cluster.cc:1428] Setting key a45f42a294482541aea2bdfab5c4880c
I20260807 22:45:47.399062 23127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221: Bootstrap starting.
I20260807 22:45:47.400023 23127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:47.400478 23127 log.cc:824] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:47.401441 23127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221: No bootstrap required, opened a new log
I20260807 22:45:47.404309 23127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } }
I20260807 22:45:47.404652 23127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:47.404779 23127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a0a6fe8314dc4a77f3e93df688221, State: Initialized, Role: FOLLOWER
I20260807 22:45:47.404999 23127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } }
I20260807 22:45:47.405164 23127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:47.405340 23127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:47.405442 23127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:47.406729 23127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } }
I20260807 22:45:47.406926 23127 leader_election.cc:302] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 059a0a6fe8314dc4a77f3e93df688221; no voters:
I20260807 22:45:47.407181 23127 leader_election.cc:288] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:47.407218 23132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:47.407416 23132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [term 1 LEADER]: Becoming Leader. State: Replica: 059a0a6fe8314dc4a77f3e93df688221, State: Running, Role: LEADER
I20260807 22:45:47.407512 23132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } }
I20260807 22:45:47.407757 23127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:47.407814 23132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } }
I20260807 22:45:47.407822 23133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 059a0a6fe8314dc4a77f3e93df688221. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } }
I20260807 22:45:47.407913 23132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:47.407918 23133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:47.408140 23136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:47.408797 23136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:47.411151 23136 catalog_manager.cc:1347] Generated new cluster ID: 9d54760e43384d9bbd11c0ae1393fd44
I20260807 22:45:47.411216 23136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 22:45:47 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
I20260807 22:45:47.419864 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:47.398691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.16.98.190:35275)
0807 22:45:47.406527 (+ 7836us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:47.406539 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:47.406723 (+ 184us) server_negotiation.cc:407] Connection header received
0807 22:45:47.406880 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:47.406888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:47.406962 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:47.407127 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:47.408104 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:47.409737 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:47.410883 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:47.411638 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:47.411723 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:47.415163 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:47.415168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:47.415168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:47.415183 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:47.417787 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:47.418333 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:47.418336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:47.418337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:47.418473 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:47.418718 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:47.418719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:47.418719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:47.419056 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:47.419204 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:47.419361 (+ 157us) server_negotiation.cc:299] Negotiation successful
0807 22:45:47.419508 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7735,"thread_start_us":124,"threads_started":1}
I20260807 22:45:47.424494 23136 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:47.425393 23136 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:47.430192 23136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221: Generated new TSK 0
I20260807 22:45:47.430426 23136 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.129.keytab.
Entry for principal oryx/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.129.keytab.
I20260807 22:45:47.465406 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:35275
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.129.keytab
--principal=oryx/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:47.559769 23153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:47.560050 23153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:47.560096 23153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:47.561466 23153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:47.561546 23153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:47.561601 23153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:45:47.563313 23153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.129.keytab
--principal=oryx/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:47.563613 23153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:47.563889 23153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:47.569826 23158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:47.569828 23159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:47.569823 23161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:47.569939 23153 server_base.cc:1034] running on GCE node
I20260807 22:45:47.570283 23153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:47.570568 23153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:47.571779 23153 hybrid_clock.cc:648] HybridClock initialized: now 1786142747571766 us; error 32 us; skew 500 ppm
Aug 07 22:45:47 dist-test-slave-rgb1 krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142747, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:47.576056 23153 init.cc:377] Logged in from keytab as oryx/127.16.98.129@KRBTEST.COM (short username oryx)
I20260807 22:45:47.576987 23153 webserver.cc:460] Webserver started at http://127.16.98.129:42059/ using document root <none> and password file <none>
I20260807 22:45:47.577296 23153 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:47.577375 23153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:47.577548 23153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:47.578745 23153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "57e9c38c98ed4690a9a434fa701ceb3a"
format_stamp: "Formatted at 2026-08-07 22:45:47 on dist-test-slave-rgb1"
server_key: "1c3dfeb335884c82c1b85ab0cd7b2e0e"
server_key_iv: "1be2d0af34e72488a539cb5f68a69dda"
server_key_version: "encryptionkey@0"
I20260807 22:45:47.579126 23153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "57e9c38c98ed4690a9a434fa701ceb3a"
format_stamp: "Formatted at 2026-08-07 22:45:47 on dist-test-slave-rgb1"
server_key: "1c3dfeb335884c82c1b85ab0cd7b2e0e"
server_key_iv: "1be2d0af34e72488a539cb5f68a69dda"
server_key_version: "encryptionkey@0"
I20260807 22:45:47.580705 23153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:45:47.581679 23169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:47.581894 23153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:47.582001 23153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "57e9c38c98ed4690a9a434fa701ceb3a"
format_stamp: "Formatted at 2026-08-07 22:45:47 on dist-test-slave-rgb1"
server_key: "1c3dfeb335884c82c1b85ab0cd7b2e0e"
server_key_iv: "1be2d0af34e72488a539cb5f68a69dda"
server_key_version: "encryptionkey@0"
I20260807 22:45:47.582089 23153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:47.591861 23153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:47.594507 23153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:47.594719 23153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:47.595091 23153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:47.595467 23153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:47.595526 23153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:47.595573 23153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:47.595605 23153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:47.602887 23153 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:42247
I20260807 22:45:47.602931 23283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:42247 every 8 connection(s)
I20260807 22:45:47.603317 23153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
Aug 07 22:45:47 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142747, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
I20260807 22:45:47.612324 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 23153
I20260807 22:45:47.612583 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:45:47.612852 16778 external_mini_cluster.cc:1428] Setting key 3617d4991fa266a8eb92709ae7510424
I20260807 22:45:47.614849 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:47.604190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:49329 (local address 127.16.98.190:35275)
0807 22:45:47.604360 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:47.604363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:47.604423 (+ 60us) server_negotiation.cc:407] Connection header received
0807 22:45:47.604989 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:47.604990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:47.605001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:47.605085 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:47.606321 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:47.606910 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:47.608315 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:47.608670 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:47.608743 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:47.611337 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:47.611338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:47.611338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:47.611346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:47.613204 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:47.613714 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:47.613715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:47.613716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:47.613808 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:47.614099 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:47.614101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:47.614101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:47.614242 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:47.614332 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:47.614645 (+ 313us) server_negotiation.cc:299] Negotiation successful
0807 22:45:47.614744 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260807 22:45:47.615089 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:47.604098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.129:49329)
0807 22:45:47.604277 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:47.604322 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:47.604883 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:47.605210 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:47.605214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:47.605920 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:47.606184 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:47.606190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:47.607061 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:47.607062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:47.608213 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:47.608216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:47.608816 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:47.608817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:47.608980 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:47.609641 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:47.609650 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:47.611243 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:47.613319 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:47.613328 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:47.613330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:47.613608 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:47.613888 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:47.613889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:47.613890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:47.614018 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:47.614323 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:47.614331 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:47.614493 (+ 162us) client_negotiation.cc:769] Sending connection context
0807 22:45:47.614621 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 22:45:47.614811 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":109,"threads_started":1}
I20260807 22:45:47.615653 23284 heartbeater.cc:347] Connected to a master server at 127.16.98.190:35275
I20260807 22:45:47.615756 23284 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:47.615980 23284 heartbeater.cc:511] Master 127.16.98.190:35275 requested a full tablet report, sending...
I20260807 22:45:47.616714 23091 ts_manager.cc:194] Registered new tserver with Master: 57e9c38c98ed4690a9a434fa701ceb3a (127.16.98.129:42247)
I20260807 22:45:47.617324 23091 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:49329
WARNING: no policy specified for oryx/127.16.98.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.130.keytab.
Entry for principal oryx/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.130.keytab.
I20260807 22:45:47.644614 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:35275
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.130.keytab
--principal=oryx/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:47.734645 23289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:47.734866 23289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:47.734894 23289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:47.736084 23289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:47.736133 23289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:47.736162 23289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:45:47.737717 23289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.130.keytab
--principal=oryx/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:47.738114 23289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:47.738396 23289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:47.744112 23295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:47.744168 23297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:47.744369 23289 server_base.cc:1034] running on GCE node
W20260807 22:45:47.744400 23294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:47.744725 23289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:47.745018 23289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:47.746228 23289 hybrid_clock.cc:648] HybridClock initialized: now 1786142747746202 us; error 45 us; skew 500 ppm
Aug 07 22:45:47 dist-test-slave-rgb1 krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142747, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:47.749316 23289 init.cc:377] Logged in from keytab as oryx/127.16.98.130@KRBTEST.COM (short username oryx)
I20260807 22:45:47.750002 23289 webserver.cc:460] Webserver started at http://127.16.98.130:40667/ using document root <none> and password file <none>
I20260807 22:45:47.750315 23289 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:47.750397 23289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:47.750617 23289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:47.751911 23289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a"
format_stamp: "Formatted at 2026-08-07 22:45:47 on dist-test-slave-rgb1"
server_key: "c5a4b7b76a003206b7ff0a2c1eb55929"
server_key_iv: "e79aadff6546969ce361ba226563a4f6"
server_key_version: "encryptionkey@0"
I20260807 22:45:47.752278 23289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a"
format_stamp: "Formatted at 2026-08-07 22:45:47 on dist-test-slave-rgb1"
server_key: "c5a4b7b76a003206b7ff0a2c1eb55929"
server_key_iv: "e79aadff6546969ce361ba226563a4f6"
server_key_version: "encryptionkey@0"
I20260807 22:45:47.753882 23289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260807 22:45:47.754858 23304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:47.755088 23289 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:47.755224 23289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a"
format_stamp: "Formatted at 2026-08-07 22:45:47 on dist-test-slave-rgb1"
server_key: "c5a4b7b76a003206b7ff0a2c1eb55929"
server_key_iv: "e79aadff6546969ce361ba226563a4f6"
server_key_version: "encryptionkey@0"
I20260807 22:45:47.755290 23289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:47.768301 23289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:47.770864 23289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:47.771055 23289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:47.771275 23289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:47.771638 23289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:47.771682 23289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:47.771704 23289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:47.771718 23289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:47.778026 23289 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:45055
I20260807 22:45:47.778126 23418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:45055 every 8 connection(s)
I20260807 22:45:47.778455 23289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:45:47.781144 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 23289
I20260807 22:45:47.781271 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:45:47.781504 16778 external_mini_cluster.cc:1428] Setting key ef8e9d9d402a182c9dd52006349f7303
Aug 07 22:45:47 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142747, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
I20260807 22:45:47.790968 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:47.779364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:36345 (local address 127.16.98.190:35275)
0807 22:45:47.779455 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:47.779457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:47.779707 (+ 250us) server_negotiation.cc:407] Connection header received
0807 22:45:47.780242 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:47.780243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:47.780254 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:47.780321 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:47.781852 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:47.782651 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:47.784348 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:47.784712 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:47.784796 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:47.787300 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:47.787301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:47.787302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:47.787314 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:47.789296 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:47.789839 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:47.789841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:47.789842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:47.789910 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:47.790202 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:47.790204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:47.790204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:47.790342 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:47.790431 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:47.790749 (+ 318us) server_negotiation.cc:299] Negotiation successful
0807 22:45:47.790848 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 22:45:47.791203 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:47.779272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.130:36345)
0807 22:45:47.779537 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:47.779583 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:47.780125 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:47.780440 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:47.780444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:47.781405 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:47.781742 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:47.781748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:47.782764 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:47.782765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:47.784229 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:47.784232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:47.784812 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:47.784814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:47.785030 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:47.785694 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:47.785704 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:47.787188 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:47.789420 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:47.789430 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:47.789461 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:47.789732 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:47.789996 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:47.789997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:47.789998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:47.790122 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:47.790421 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:47.790429 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:47.790595 (+ 166us) client_negotiation.cc:769] Sending connection context
0807 22:45:47.790724 (+ 129us) client_negotiation.cc:240] Negotiation successful
0807 22:45:47.790920 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for oryx/127.16.98.131@KRBTEST.COM; defaulting to no policy
I20260807 22:45:47.792052 23419 heartbeater.cc:347] Connected to a master server at 127.16.98.190:35275
I20260807 22:45:47.792179 23419 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:47.792398 23419 heartbeater.cc:511] Master 127.16.98.190:35275 requested a full tablet report, sending...
I20260807 22:45:47.793072 23091 ts_manager.cc:194] Registered new tserver with Master: f89601ecd4ae42c59e1ffbd5bd25ad1a (127.16.98.130:45055)
I20260807 22:45:47.793553 23091 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:36345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.131.keytab.
Entry for principal oryx/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.131.keytab.
I20260807 22:45:47.816253 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:35275
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.131.keytab
--principal=oryx/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:47.908588 23424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:47.908874 23424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:47.908929 23424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:47.910349 23424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:47.910403 23424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:47.910434 23424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:45:47.912021 23424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.131.keytab
--principal=oryx/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:35275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:47.912285 23424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:47.912597 23424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:47.918322 23430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:47.918282 23431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:47.918632 23424 server_base.cc:1034] running on GCE node
W20260807 22:45:47.918282 23433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:47.918969 23424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:47.919286 23424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:47.920499 23424 hybrid_clock.cc:648] HybridClock initialized: now 1786142747920485 us; error 89 us; skew 500 ppm
Aug 07 22:45:47 dist-test-slave-rgb1 krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142747, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:47.923723 23424 init.cc:377] Logged in from keytab as oryx/127.16.98.131@KRBTEST.COM (short username oryx)
I20260807 22:45:47.924289 23424 webserver.cc:460] Webserver started at http://127.16.98.131:43989/ using document root <none> and password file <none>
I20260807 22:45:47.924649 23424 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:47.924723 23424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:47.924870 23424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:47.925987 23424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "93be69dbc9034b229bbc074bc8fde968"
format_stamp: "Formatted at 2026-08-07 22:45:47 on dist-test-slave-rgb1"
server_key: "e522c015eb28ac0ebc3d49e56c5e8708"
server_key_iv: "1e61a540494cb961d6f91a06c1bede21"
server_key_version: "encryptionkey@0"
I20260807 22:45:47.926370 23424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "93be69dbc9034b229bbc074bc8fde968"
format_stamp: "Formatted at 2026-08-07 22:45:47 on dist-test-slave-rgb1"
server_key: "e522c015eb28ac0ebc3d49e56c5e8708"
server_key_iv: "1e61a540494cb961d6f91a06c1bede21"
server_key_version: "encryptionkey@0"
I20260807 22:45:47.927802 23424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260807 22:45:47.928794 23441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:47.929060 23424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 22:45:47.929171 23424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "93be69dbc9034b229bbc074bc8fde968"
format_stamp: "Formatted at 2026-08-07 22:45:47 on dist-test-slave-rgb1"
server_key: "e522c015eb28ac0ebc3d49e56c5e8708"
server_key_iv: "1e61a540494cb961d6f91a06c1bede21"
server_key_version: "encryptionkey@0"
I20260807 22:45:47.929261 23424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:47.936734 23424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:47.939340 23424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:47.939538 23424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:47.939834 23424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:47.940187 23424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:47.940244 23424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:47.940289 23424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:47.940317 23424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:47.948787 23424 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:43797
I20260807 22:45:47.948882 23556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:43797 every 8 connection(s)
I20260807 22:45:47.949249 23424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:45:47.953181 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 23424
I20260807 22:45:47.953296 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:45:47.953534 16778 external_mini_cluster.cc:1428] Setting key cf08ea3fc1028624961763cf4674ad22
Aug 07 22:45:47 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142747, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
I20260807 22:45:47.962069 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:47.950586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:37171 (local address 127.16.98.190:35275)
0807 22:45:47.950640 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:47.950643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:47.950655 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:45:47.951294 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:47.951296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:47.951311 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:47.951383 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:47.953382 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:47.954095 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:47.955211 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:47.955428 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:47.955770 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:47.958604 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:47.958605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:47.958605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:47.958615 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:47.960191 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:47.960758 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:47.960761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:47.960761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:47.960899 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:47.961341 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:47.961342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:47.961342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:47.961497 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:47.961564 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:47.961906 (+ 342us) server_negotiation.cc:299] Negotiation successful
0807 22:45:47.961986 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260807 22:45:47.962301 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:47.950289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.131:37171)
0807 22:45:47.950490 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:47.950543 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:47.951151 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:47.951488 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:47.951492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:47.952450 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:47.953122 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:47.953132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:47.954231 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:47.954233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:47.955098 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:47.955100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:47.955785 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:47.955787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:47.955949 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:47.956613 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:47.956624 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:47.958476 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:47.960293 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:47.960300 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:47.960303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:47.960617 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:47.961042 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:47.961043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:47.961044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:47.961196 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:47.961595 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:47.961602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:47.961741 (+ 139us) client_negotiation.cc:769] Sending connection context
0807 22:45:47.961837 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 22:45:47.962046 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":87,"threads_started":1}
I20260807 22:45:47.963133 23557 heartbeater.cc:347] Connected to a master server at 127.16.98.190:35275
I20260807 22:45:47.963449 23557 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:47.963765 23557 heartbeater.cc:511] Master 127.16.98.190:35275 requested a full tablet report, sending...
I20260807 22:45:47.964545 23091 ts_manager.cc:194] Registered new tserver with Master: 93be69dbc9034b229bbc074bc8fde968 (127.16.98.131:43797)
I20260807 22:45:47.965265 23091 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:37171
I20260807 22:45:47.965796 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.98.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.98.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.98.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.189.keytab.
Entry for principal oryx/127.16.98.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.189.keytab.
I20260807 22:45:48.006183 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.189:36023
--webserver_interface=127.16.98.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.189.keytab
--principal=oryx/127.16.98.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:35275,127.16.98.189:36023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:48.095340 23562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:48.095604 23562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:48.095646 23562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:48.096906 23562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:48.096968 23562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:48.096994 23562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:48.097041 23562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:48.097069 23562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:48.098600 23562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:35275,127.16.98.189:36023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.189.keytab
--principal=oryx/127.16.98.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.189:36023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.98.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
W20260807 22:45:48.098752 23562 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.98.190:35275,127.16.98.189:36023'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260807 22:45:48.098910 23562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:48.099144 23562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:48.104545 23568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:48.104544 23567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:48.104542 23570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:48.104818 23562 server_base.cc:1034] running on GCE node
I20260807 22:45:48.105132 23562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:48.105463 23562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:48.106768 23562 hybrid_clock.cc:648] HybridClock initialized: now 1786142748106753 us; error 34 us; skew 500 ppm
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:48.109696 23562 init.cc:377] Logged in from keytab as oryx/127.16.98.189@KRBTEST.COM (short username oryx)
I20260807 22:45:48.110203 23562 webserver.cc:460] Webserver started at http://127.16.98.189:39951/ using document root <none> and password file <none>
I20260807 22:45:48.110437 23562 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:48.110488 23562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:48.110623 23562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:48.111730 23562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/data/instance:
uuid: "9b7eb91351014fcd878bb334667f3c43"
format_stamp: "Formatted at 2026-08-07 22:45:48 on dist-test-slave-rgb1"
server_key: "4f5a07fcad2b664269ee2e6a468c9c6a"
server_key_iv: "0dca507f3989626661606233f915c5db"
server_key_version: "encryptionkey@0"
I20260807 22:45:48.112100 23562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/wal/instance:
uuid: "9b7eb91351014fcd878bb334667f3c43"
format_stamp: "Formatted at 2026-08-07 22:45:48 on dist-test-slave-rgb1"
server_key: "4f5a07fcad2b664269ee2e6a468c9c6a"
server_key_iv: "0dca507f3989626661606233f915c5db"
server_key_version: "encryptionkey@0"
I20260807 22:45:48.113583 23562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:45:48.114683 23577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:48.115145 23562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:48.115406 23562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/wal
uuid: "9b7eb91351014fcd878bb334667f3c43"
format_stamp: "Formatted at 2026-08-07 22:45:48 on dist-test-slave-rgb1"
server_key: "4f5a07fcad2b664269ee2e6a468c9c6a"
server_key_iv: "0dca507f3989626661606233f915c5db"
server_key_version: "encryptionkey@0"
I20260807 22:45:48.115552 23562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:48.146593 23562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:48.149497 23562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:48.149642 23562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:48.154886 23562 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.189:36023
I20260807 22:45:48.154883 23630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.189:36023 every 8 connection(s)
I20260807 22:45:48.155869 23562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/data/info.pb
I20260807 22:45:48.157483 23631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:48.159569 23631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:45:48.163694 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 23562
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.189@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.98.189@KRBTEST.COM
I20260807 22:45:48.172669 23635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.160386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:38956)
0807 22:45:48.160684 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.160734 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.161261 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.161632 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.161636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.161951 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.162245 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.162250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.163249 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.163250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.164486 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.164490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.165164 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.165165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.165314 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.166039 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.166064 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.167949 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.170157 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.170163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.170165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.170646 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.171095 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.171096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.171097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.171259 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.171808 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.171815 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.172131 (+ 316us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.172267 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.172405 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":183,"threads_started":1}
I20260807 22:45:48.172664 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.160467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.16.98.190:35275)
0807 22:45:48.160550 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.160552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.160846 (+ 294us) server_negotiation.cc:407] Connection header received
0807 22:45:48.161411 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.161412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.161423 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.161502 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.162375 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.163008 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.164618 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.164991 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.165084 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.168115 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.168117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.168117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.168135 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.170043 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.170834 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.170837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.170837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.170964 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.171357 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.171359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.171359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.171652 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.171735 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.172422 (+ 687us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.172537 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 22:45:48.173417 23631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:45:48.177278 23637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.164376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36174 (local address 127.16.98.189:36023)
0807 22:45:48.164627 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.164634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.164693 (+ 59us) server_negotiation.cc:407] Connection header received
0807 22:45:48.164747 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.164749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.164767 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.164834 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.165646 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.167071 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.168091 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.168459 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.168533 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.171910 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.171916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.171917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.171925 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.174955 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.175551 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.175554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.175554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.175617 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.176004 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.176006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.176007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.176483 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.176715 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.177112 (+ 397us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.177196 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":118,"threads_started":1}
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.189@KRBTEST.COM for oryx/127.16.98.189@KRBTEST.COM
I20260807 22:45:48.182632 23635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.173751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:36023 (local address 127.0.0.1:36180)
0807 22:45:48.173787 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.173798 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.173841 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.174371 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.174372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.174504 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.174564 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.174565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.175503 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.175505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.176249 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.176252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.176918 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.176920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.177082 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.177802 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.177810 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.179185 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.181465 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.181466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.181466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.181636 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.181850 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.181851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.181851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.181907 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.182222 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.182223 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.182262 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.182331 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.182533 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":103}
I20260807 22:45:48.182632 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.173791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36180 (local address 127.16.98.189:36023)
0807 22:45:48.174060 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.174063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.174079 (+ 16us) server_negotiation.cc:407] Connection header received
0807 22:45:48.174114 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.174115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.174131 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.174390 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.174656 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.175388 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.176435 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.176822 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.176891 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.179320 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.179321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.179322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.179328 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.181370 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.181722 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.181723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.181724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.181783 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.182036 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.182037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.182037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.182147 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.182231 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.182359 (+ 128us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.182453 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":109,"threads_started":1}
I20260807 22:45:48.184160 23631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: Bootstrap starting.
I20260807 22:45:48.184849 23631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:48.185141 23631 log.cc:824] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:48.186154 23631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: No bootstrap required, opened a new log
I20260807 22:45:48.187753 23631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } }
I20260807 22:45:48.187959 23631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:48.188042 23631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b7eb91351014fcd878bb334667f3c43, State: Initialized, Role: FOLLOWER
I20260807 22:45:48.188247 23631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } }
I20260807 22:45:48.188609 23640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } } }
I20260807 22:45:48.188727 23631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:48.188727 23640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 22:45:48.190992 23651 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:45:48.191082 23651 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:45:48.193449 23562 master_runner.cc:428] Detected that this master 9b7eb91351014fcd878bb334667f3c43 is joining an existing cluster
I20260807 22:45:48.193549 23562 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.98.189:36023
I20260807 22:45:48.201751 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.194776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.16.98.190:35275)
0807 22:45:48.194854 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.194856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.194996 (+ 140us) server_negotiation.cc:407] Connection header received
0807 22:45:48.195169 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.195170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.195182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.195259 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.195715 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.196235 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.197067 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.197273 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.197360 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.198526 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.198528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.198528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.198536 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.200315 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.200818 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.200820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.200820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.200905 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.201138 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.201139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.201139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.201292 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.201370 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.201573 (+ 203us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.201656 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 22:45:48.201965 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.194697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:38964)
0807 22:45:48.194930 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.194942 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.195079 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.195269 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.195270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.195525 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.195632 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.195634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.196321 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.196322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.196973 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.196975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.197390 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.197391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.197452 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.197979 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.197985 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.198439 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.200405 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.200407 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.200408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.200734 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.200987 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.200988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.200988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.201061 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.201374 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.201375 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.201427 (+ 52us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.201530 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.201647 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":118,"threads_started":1}
I20260807 22:45:48.202852 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.194972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:36023 (local address 127.0.0.1:36194)
0807 22:45:48.195142 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.195151 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.195226 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.195383 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.195383 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.195640 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.195701 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.195703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.196668 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.196668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.197492 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.197494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.198341 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.198342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.198415 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.198912 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.198916 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.199330 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.201191 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.201191 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.201192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.201407 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.201610 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.201611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.201611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.201665 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.202130 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.202130 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.202166 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.202263 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.202689 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":92,"threads_started":1}
I20260807 22:45:48.203029 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.195036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.16.98.189:36023)
0807 22:45:48.195087 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.195090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.195205 (+ 115us) server_negotiation.cc:407] Connection header received
0807 22:45:48.195290 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.195291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.195300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.195329 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.195791 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.196544 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.197579 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.198230 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.198306 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.199411 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.199413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.199413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.199420 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.201126 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.201471 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.201472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.201473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.201537 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.201866 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.201867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.201868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.202024 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.202078 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.202306 (+ 228us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.202846 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":431,"server-negotiator.queue_time_us":31}
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142747, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.190@KRBTEST.COM for oryx/127.16.98.189@KRBTEST.COM
I20260807 22:45:48.216094 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.205912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.16.98.189:36023)
0807 22:45:48.205974 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.205976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.206184 (+ 208us) server_negotiation.cc:407] Connection header received
0807 22:45:48.206265 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.206266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.206278 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.206342 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.207082 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.207905 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.209174 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.209517 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.209600 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.212040 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.212042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.212042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.212050 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.214492 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.215022 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.215023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.215024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.215086 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.215355 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.215356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.215357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.215571 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.215657 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.215889 (+ 232us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.215966 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 22:45:48.216107 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.205758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:36023 (local address 127.0.0.1:36196)
0807 22:45:48.206000 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.206112 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.206185 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.206418 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.206428 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.206844 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.206952 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.206955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.208025 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.208027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.209079 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.209082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.209620 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.209621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.209756 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.210350 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.210354 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.211943 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.214609 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.214620 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.214623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.214921 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.215167 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.215168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.215169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.215243 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.215665 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.215674 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.215737 (+ 63us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.215839 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.215996 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":125,"threads_started":1}
I20260807 22:45:48.217293 23091 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.98.189:36023: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9b7eb91351014fcd878bb334667f3c43"
member_type: NON_VOTER
last_known_addr {
host: "127.16.98.189"
port: 36023
}
attrs {
promote: true
}
}
dest_uuid: "059a0a6fe8314dc4a77f3e93df688221"
cas_config_opid_index: -1
I20260807 22:45:48.217660 23091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: NON_VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: true } }
I20260807 22:45:48.218237 23662 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9b7eb91351014fcd878bb334667f3c43 (127.16.98.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: NON_VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: true } } }
I20260807 22:45:48.218609 23604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:48.219365 23662 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.98.189:36023
I20260807 22:45:48.219571 23661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: NON_VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: true } } }
I20260807 22:45:48.219765 23604 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 FOLLOWER]: Refusing update from remote peer 059a0a6fe8314dc4a77f3e93df688221: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260807 22:45:48.219766 23661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:48.219812 23562 master.cc:537] Master@127.16.98.189:36023 shutting down...
I20260807 22:45:48.220429 23661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: NON_VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 22:45:48.221474 23074 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260807 22:45:48.222103 23074 consensus_peers.cc:487] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 -> Peer 9b7eb91351014fcd878bb334667f3c43 (127.16.98.189:36023): Couldn't send request to peer 9b7eb91351014fcd878bb334667f3c43. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 22:45:48.223146 23562 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:48.223248 23562 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:48.223325 23562 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: stopping tablet replica
I20260807 22:45:48.236668 23562 master.cc:552] Master@127.16.98.189:36023 shutdown complete.
I20260807 22:45:48.236727 23562 master_runner.cc:305] Clearing existing system tablet
I20260807 22:45:48.236884 23562 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:48.236938 23562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:48.237697 23562 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:48.238739 23666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:48.239013 23562 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:48.239111 23562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:48.239147 23562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/wal
uuid: "9b7eb91351014fcd878bb334667f3c43"
format_stamp: "Formatted at 2026-08-07 22:45:48 on dist-test-slave-rgb1"
server_key: "4f5a07fcad2b664269ee2e6a468c9c6a"
server_key_iv: "0dca507f3989626661606233f915c5db"
server_key_version: "encryptionkey@0"
I20260807 22:45:48.239877 23562 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:48.241628 23562 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 22:45:48.241720 23562 log.cc:1196] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260807 22:45:48.241961 23562 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: Deleting consensus metadata
I20260807 22:45:48.242105 23562 master_runner.cc:315] Copying system tablet from 127.16.98.190:35275
I20260807 22:45:48.242790 23562 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: tablet copy: Beginning tablet copy session from remote peer at address 127.16.98.190:35275
I20260807 22:45:48.249802 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.243244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38976 (local address 127.16.98.190:35275)
0807 22:45:48.243300 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.243302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.243381 (+ 79us) server_negotiation.cc:407] Connection header received
0807 22:45:48.243457 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.243458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.243470 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.243528 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.243967 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.244674 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.245745 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.245953 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.246024 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.247137 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.247138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.247138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.247145 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.248706 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.249091 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.249092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.249092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.249146 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.249341 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.249342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.249342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.249446 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.249508 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.249669 (+ 161us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.249735 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260807 22:45:48.249866 23672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.243141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:38976)
0807 22:45:48.243296 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.243309 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.243385 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.243553 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.243555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.243784 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.243862 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.243864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.244767 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.244768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.245643 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.245646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.246045 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.246046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.246116 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.246640 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.246645 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.247048 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.248783 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.248784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.248785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.249013 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.249213 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.249214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.249214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.249280 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.249517 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.249518 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.249566 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.249639 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.249748 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260807 22:45:48.250567 23111 tablet_copy_service.cc:140] P 059a0a6fe8314dc4a77f3e93df688221: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9b7eb91351014fcd878bb334667f3c43 ({username='oryx', principal='oryx/127.16.98.189@KRBTEST.COM'} at 127.0.0.1:38976)
I20260807 22:45:48.250658 23111 tablet_copy_service.cc:161] P 059a0a6fe8314dc4a77f3e93df688221: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9b7eb91351014fcd878bb334667f3c43 at {username='oryx', principal='oryx/127.16.98.189@KRBTEST.COM'} at 127.0.0.1:38976: session id = 9b7eb91351014fcd878bb334667f3c43-00000000000000000000000000000000
I20260807 22:45:48.251358 23111 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 22:45:48.251988 23562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 22:45:48.253199 23562 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: tablet copy: Starting download of 0 data blocks...
I20260807 22:45:48.253353 23562 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: tablet copy: Starting download of 1 WAL segments...
I20260807 22:45:48.254374 23562 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 22:45:48.255498 23111 tablet_copy_service.cc:342] P 059a0a6fe8314dc4a77f3e93df688221: Request end of tablet copy session 9b7eb91351014fcd878bb334667f3c43-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.98.189@KRBTEST.COM'} at 127.0.0.1:38976
I20260807 22:45:48.255561 23111 tablet_copy_service.cc:433] P 059a0a6fe8314dc4a77f3e93df688221: ending tablet copy session 9b7eb91351014fcd878bb334667f3c43-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9b7eb91351014fcd878bb334667f3c43
I20260807 22:45:48.258890 23562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:48.260224 23673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:48.260293 23676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:48.260391 23562 server_base.cc:1034] running on GCE node
W20260807 22:45:48.260303 23674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:48.260685 23562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:48.260854 23562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:48.268206 23562 hybrid_clock.cc:648] HybridClock initialized: now 1786142748268182 us; error 52 us; skew 500 ppm
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:48.269979 23562 init.cc:377] Logged in from keytab as oryx/127.16.98.189@KRBTEST.COM (short username oryx)
I20260807 22:45:48.270368 23562 webserver.cc:460] Webserver started at http://127.16.98.189:42555/ using document root <none> and password file <none>
I20260807 22:45:48.270536 23562 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:48.270589 23562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:48.271384 23562 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:48.272076 23683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:48.272229 23562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:48.272287 23562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/wal
uuid: "9b7eb91351014fcd878bb334667f3c43"
format_stamp: "Formatted at 2026-08-07 22:45:48 on dist-test-slave-rgb1"
server_key: "4f5a07fcad2b664269ee2e6a468c9c6a"
server_key_iv: "0dca507f3989626661606233f915c5db"
server_key_version: "encryptionkey@0"
I20260807 22:45:48.272362 23562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:48.291234 23562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:48.293444 23562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:48.297420 23562 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.189:36023
I20260807 22:45:48.297437 23735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.189:36023 every 8 connection(s)
I20260807 22:45:48.297807 23562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-1/data/info.pb
I20260807 22:45:48.298712 23736 sys_catalog.cc:263] Verifying existing consensus state
I20260807 22:45:48.299263 23736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: Bootstrap starting.
I20260807 22:45:48.302546 23736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:45:48.302796 23736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: Bootstrap complete.
I20260807 22:45:48.303030 23736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: NON_VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: true } }
I20260807 22:45:48.303251 23736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b7eb91351014fcd878bb334667f3c43, State: Initialized, Role: LEARNER
I20260807 22:45:48.303337 23736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: NON_VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: true } }
I20260807 22:45:48.303620 23739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: NON_VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: true } } }
I20260807 22:45:48.303663 23736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:48.303742 23739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: This master's current role is: LEARNER
I20260807 22:45:48.305430 23750 catalog_manager.cc:1259] Loaded cluster ID: 9d54760e43384d9bbd11c0ae1393fd44
I20260807 22:45:48.305481 23750 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: loading cluster ID for follower catalog manager: success
I20260807 22:45:48.306509 23750 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: acquiring CA information for follower catalog manager: success
I20260807 22:45:48.306792 23750 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 22:45:48.337666 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.330102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36216 (local address 127.16.98.189:36023)
0807 22:45:48.330284 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.330286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.330293 (+ 7us) server_negotiation.cc:407] Connection header received
0807 22:45:48.330316 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.330317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.330326 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.330401 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.330981 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.331655 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.332511 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.332817 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.332896 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.334333 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.334334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.334334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.334342 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.336163 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.336723 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.336725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.336726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.336794 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.337035 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.337036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.337036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.337168 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.337287 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.337423 (+ 136us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.337521 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for oryx/127.16.98.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.98.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.98.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.188.keytab.
Entry for principal oryx/127.16.98.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.188.keytab.
I20260807 22:45:48.374426 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.188:35661
--webserver_interface=127.16.98.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.188.keytab
--principal=oryx/127.16.98.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:35275,127.16.98.189:36023,127.16.98.188:35661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:48.465420 23754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:48.465672 23754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:48.465696 23754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:48.466871 23754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:48.466918 23754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:48.466933 23754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:48.466946 23754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:48.466959 23754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:48.468735 23754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:35275,127.16.98.189:36023,127.16.98.188:35661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.188.keytab
--principal=oryx/127.16.98.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.188:35661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.98.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:48.468972 23754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:48.469175 23754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:48.475432 23760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:48.475469 23762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:48.475428 23759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:48.476472 23754 server_base.cc:1034] running on GCE node
I20260807 22:45:48.476840 23754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:48.477200 23754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:48.478382 23754 hybrid_clock.cc:648] HybridClock initialized: now 1786142748478334 us; error 74 us; skew 500 ppm
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:48.482112 23754 init.cc:377] Logged in from keytab as oryx/127.16.98.188@KRBTEST.COM (short username oryx)
I20260807 22:45:48.482656 23754 webserver.cc:460] Webserver started at http://127.16.98.188:43653/ using document root <none> and password file <none>
I20260807 22:45:48.482919 23754 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:48.482996 23754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:48.483225 23754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:48.484373 23754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/data/instance:
uuid: "0cb3166b76864a979588c5abf24a5ba8"
format_stamp: "Formatted at 2026-08-07 22:45:48 on dist-test-slave-rgb1"
server_key: "afb11465cdb4ca64604abbd908837025"
server_key_iv: "9ee4e3c312de0634612ab8e29d5fff71"
server_key_version: "encryptionkey@0"
I20260807 22:45:48.484805 23754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/wal/instance:
uuid: "0cb3166b76864a979588c5abf24a5ba8"
format_stamp: "Formatted at 2026-08-07 22:45:48 on dist-test-slave-rgb1"
server_key: "afb11465cdb4ca64604abbd908837025"
server_key_iv: "9ee4e3c312de0634612ab8e29d5fff71"
server_key_version: "encryptionkey@0"
I20260807 22:45:48.486449 23754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:48.487675 23769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:48.488045 23754 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:48.488273 23754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/wal
uuid: "0cb3166b76864a979588c5abf24a5ba8"
format_stamp: "Formatted at 2026-08-07 22:45:48 on dist-test-slave-rgb1"
server_key: "afb11465cdb4ca64604abbd908837025"
server_key_iv: "9ee4e3c312de0634612ab8e29d5fff71"
server_key_version: "encryptionkey@0"
I20260807 22:45:48.488406 23754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:48.498517 23754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:48.501381 23754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:48.501580 23754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:48.507102 23754 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.188:35661
I20260807 22:45:48.507138 23822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.188:35661 every 8 connection(s)
I20260807 22:45:48.507611 23754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/data/info.pb
I20260807 22:45:48.508927 23823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:48.510636 23823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:45:48.511418 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 23754
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.188@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.98.188@KRBTEST.COM
I20260807 22:45:48.523141 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.511519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38988 (local address 127.16.98.190:35275)
0807 22:45:48.511579 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.511581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.511786 (+ 205us) server_negotiation.cc:407] Connection header received
0807 22:45:48.512438 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.512439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.512453 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.512516 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.513144 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.513732 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.515360 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.515609 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.515868 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.519060 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.519062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.519063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.519074 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.521222 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.521971 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.521973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.521973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.522057 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.522479 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.522481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.522481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.522643 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.522743 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.522948 (+ 205us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.523046 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 22:45:48.523339 23825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.511458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:38988)
0807 22:45:48.511654 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.511702 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.512319 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.512539 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.512543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.512882 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.513051 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.513056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.514001 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.514003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.515246 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.515249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.515946 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.515947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.516084 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.516883 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.516893 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.518819 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.521412 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.521432 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.521436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.521861 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.522215 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.522216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.522217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.522374 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.522735 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.522742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.522819 (+ 77us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.522920 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.523056 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":98,"threads_started":1}
I20260807 22:45:48.523344 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.512199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56190 (local address 127.16.98.188:35661)
0807 22:45:48.512381 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.512386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.512416 (+ 30us) server_negotiation.cc:407] Connection header received
0807 22:45:48.512485 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.512486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.512503 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.512736 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.513405 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.514465 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.515715 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.516154 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.516257 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.519359 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.519367 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.519368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.519375 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.521426 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.521865 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.521869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.521870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.521975 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.522240 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.522241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.522241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.522545 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.522737 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.522833 (+ 96us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.522992 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260807 22:45:48.523960 23823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.188@KRBTEST.COM for oryx/127.16.98.189@KRBTEST.COM
I20260807 22:45:48.532922 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.524417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.16.98.189:36023)
0807 22:45:48.524548 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.524550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.524619 (+ 69us) server_negotiation.cc:407] Connection header received
0807 22:45:48.524656 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.524664 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.524677 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.524735 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.525151 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.525702 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.526446 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.526655 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.526730 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.529181 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.529183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.529183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.529191 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.530961 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.531490 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.531491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.531491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.531558 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.531935 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.531936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.531936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.532135 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.532212 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.532698 (+ 486us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.532789 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 22:45:48.533172 23825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.524283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:36023 (local address 127.0.0.1:36220)
0807 22:45:48.524406 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.524427 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.524486 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.524787 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.524788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.524955 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.524995 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.524997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.525800 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.525801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.526334 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.526337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.526769 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.526770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.526823 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.527466 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.527471 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.529076 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.531136 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.531138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.531138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.531373 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.531690 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.531691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.531692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.531802 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.532371 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.532373 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.532457 (+ 84us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.532642 (+ 185us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.533018 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260807 22:45:48.533716 23823 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.188@KRBTEST.COM for oryx/127.16.98.188@KRBTEST.COM
I20260807 22:45:48.541383 23825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.534242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:35661 (local address 127.0.0.1:56206)
0807 22:45:48.534336 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.534351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.534416 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.534587 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.534588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.534718 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.534763 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.534765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.535503 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.535504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.536066 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.536068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.536451 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.536452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.536493 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.537153 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.537157 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.538386 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.540116 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.540117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.540118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.540279 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.540524 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.540529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.540530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.540649 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.540974 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.540975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.541083 (+ 108us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.541161 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.541248 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260807 22:45:48.541399 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.534366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.16.98.188:35661)
0807 22:45:48.534425 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.534427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.534438 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:45:48.534490 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.534491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.534506 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.534571 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.534870 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.535382 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.536161 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.536362 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.536447 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.538475 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.538477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.538477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.538483 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.540030 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.540382 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.540383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.540383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.540442 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.540740 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.540741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.540741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.540882 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.540943 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.541226 (+ 283us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.541298 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260807 22:45:48.544138 23823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: Bootstrap starting.
I20260807 22:45:48.545432 23823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:48.546123 23823 log.cc:824] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:48.547351 23823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: No bootstrap required, opened a new log
I20260807 22:45:48.549512 23823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } }
I20260807 22:45:48.549714 23823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:48.549759 23823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb3166b76864a979588c5abf24a5ba8, State: Initialized, Role: FOLLOWER
I20260807 22:45:48.549949 23823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } }
I20260807 22:45:48.550329 23823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:48.550293 23828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } } }
I20260807 22:45:48.550408 23828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 22:45:48.552451 23840 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:45:48.552647 23840 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:45:48.557871 23754 master_runner.cc:428] Detected that this master 0cb3166b76864a979588c5abf24a5ba8 is joining an existing cluster
I20260807 22:45:48.557960 23754 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.98.188:35661
I20260807 22:45:48.569283 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.559693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38996 (local address 127.16.98.190:35275)
0807 22:45:48.559744 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.559746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.559915 (+ 169us) server_negotiation.cc:407] Connection header received
0807 22:45:48.559945 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.559946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.559957 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.560009 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.560628 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.561379 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.563107 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.563510 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.563557 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.565593 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.565595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.565595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.565605 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.567825 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.568328 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.568330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.568330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.568406 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.568684 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.568685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.568685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.568818 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.568856 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.569086 (+ 230us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.569182 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 22:45:48.569321 23847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.559497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:38996)
0807 22:45:48.559730 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.559742 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.559815 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.560080 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.560081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.560424 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.560505 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.560507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.561476 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.561477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.562827 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.562836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.563645 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.563646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.563707 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.564965 (+ 1258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.564970 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.565451 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.567999 (+ 2548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.568001 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.568001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.568226 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.568486 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.568487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.568487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.568595 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.568906 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.568907 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.568963 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.569038 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.569178 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":102,"thread_start_us":180,"threads_started":1}
I20260807 22:45:48.575872 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 23153
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142747, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.190@KRBTEST.COM for oryx/127.16.98.188@KRBTEST.COM
I20260807 22:45:48.585896 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:42247
--local_ip_for_outbound_sockets=127.16.98.129
--tserver_master_addrs=127.16.98.190:35275
--webserver_port=42059
--webserver_interface=127.16.98.129
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.129.keytab
--principal=oryx/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.98.190:35275,127.16.98.189:36023
--tserver_master_addrs=127.16.98.190:35275,127.16.98.189:36023,127.16.98.188:35661 with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:45:48.586061 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.559507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:35661 (local address 127.0.0.1:56210)
0807 22:45:48.559915 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.559926 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.559995 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.560141 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.560141 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.560415 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.560510 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.560511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.561476 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.561476 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.562827 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.562836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.563611 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.563612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.563677 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.581545 (+ 17868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.581551 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.582208 (+ 657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.584798 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.584799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.584800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.585031 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.585251 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.585252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.585252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.585332 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.585679 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.585680 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.585723 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.585796 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.585924 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":106,"thread_start_us":102,"threads_started":1}
I20260807 22:45:48.586272 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.573285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:35661 (local address 127.0.0.1:56222)
0807 22:45:48.573350 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.573362 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.573407 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.573873 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.573874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.574057 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.574107 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.574109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.575651 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.575653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.577044 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.577056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.577644 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.577645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.577721 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.578283 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.578289 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.579727 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.584877 (+ 5150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.584878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.584879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.585166 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.585432 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.585433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.585434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.585497 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.585794 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.585795 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.585847 (+ 52us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.585903 (+ 56us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.586016 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260807 22:45:48.586422 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.559588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.16.98.188:35661)
0807 22:45:48.559638 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.559641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.559975 (+ 334us) server_negotiation.cc:407] Connection header received
0807 22:45:48.560044 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.560045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.560055 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.560102 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.560625 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.561378 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.563104 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.563418 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.563474 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.582357 (+ 18883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.582359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.582359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.582368 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.584685 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.585121 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.585123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.585123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.585180 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.585407 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.585408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.585408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.585588 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.585656 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.586201 (+ 545us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.586309 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 22:45:48.586650 23849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.573491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.16.98.188:35661)
0807 22:45:48.573691 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.573693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.573701 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:45:48.573744 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.573745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.573772 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.573849 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.574189 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.575494 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.577148 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.577552 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.577921 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.579876 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.579878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.579879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.579887 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.584758 (+ 4871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.585263 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.585264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.585265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.585342 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.585579 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.585579 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.585580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.585704 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.586456 (+ 752us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.586501 (+ 45us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.586584 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":179,"thread_start_us":145,"threads_started":1}
I20260807 22:45:48.587260 23091 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.98.188:35661: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8"
member_type: NON_VOTER
last_known_addr {
host: "127.16.98.188"
port: 35661
}
attrs {
promote: true
}
}
dest_uuid: "059a0a6fe8314dc4a77f3e93df688221"
cas_config_opid_index: 5
I20260807 22:45:48.587610 23091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: NON_VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: true } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } }
I20260807 22:45:48.588893 23662 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0cb3166b76864a979588c5abf24a5ba8 (127.16.98.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: NON_VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: true } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } } }
I20260807 22:45:48.590121 23662 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.98.188:35661
I20260807 22:45:48.590211 23662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: NON_VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: true } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } } }
I20260807 22:45:48.590317 23662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:48.590613 23754 master.cc:537] Master@127.16.98.188:35661 shutting down...
W20260807 22:45:48.591442 23075 consensus_peers.cc:487] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 -> Peer 0cb3166b76864a979588c5abf24a5ba8 (127.16.98.188:35661): Couldn't send request to peer 0cb3166b76864a979588c5abf24a5ba8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 22:45:48.596158 23754 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:48.596280 23754 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:48.596342 23754 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: stopping tablet replica
I20260807 22:45:48.596985 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.559872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36222 (local address 127.16.98.189:36023)
0807 22:45:48.559916 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.559918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.560138 (+ 220us) server_negotiation.cc:407] Connection header received
0807 22:45:48.560235 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.560237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.560248 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.560279 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.560800 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.561659 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.563657 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.564079 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.564142 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.592041 (+ 27899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.592043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.592043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.592052 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.594479 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.595620 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.595621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.595621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.595679 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.596179 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.596180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.596180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.596333 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.596377 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.596790 (+ 413us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.596885 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260807 22:45:48.600694 23754 master.cc:552] Master@127.16.98.188:35661 shutdown complete.
I20260807 22:45:48.600761 23754 master_runner.cc:305] Clearing existing system tablet
I20260807 22:45:48.600927 23754 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:48.600991 23754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:48.602052 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.590637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:36023 (local address 127.0.0.1:36236)
0807 22:45:48.590871 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.590884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.590936 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.597402 (+ 6466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.597403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.597406 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:48.597462 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.597464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.599364 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.599365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.600706 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.600711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.601767 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.601768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.601841 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.601865 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.601919 (+ 54us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.601969 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20260807 22:45:48.602480 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.591077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36236 (local address 127.16.98.189:36023)
0807 22:45:48.597067 (+ 5990us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.597069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.597077 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:45:48.597097 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.597098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.597125 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.597194 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:48.597673 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.598430 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.600896 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.601518 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.601593 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.601664 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.602352 (+ 688us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.602401 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5974,"thread_start_us":80,"threads_started":1}
I20260807 22:45:48.602852 23710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 LEARNER]: Refusing update from remote peer 059a0a6fe8314dc4a77f3e93df688221: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 22:45:48.603170 23851 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: NON_VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 22:45:48.603904 23710 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0cb3166b76864a979588c5abf24a5ba8 (127.16.98.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: NON_VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: true } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } } }
I20260807 22:45:48.605746 23739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 059a0a6fe8314dc4a77f3e93df688221. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: NON_VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: true } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } } }
I20260807 22:45:48.605841 23739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: This master's current role is: LEARNER
I20260807 22:45:48.605877 23739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: NON_VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: true } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } } }
I20260807 22:45:48.605921 23739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: This master's current role is: LEARNER
I20260807 22:45:48.603451 23848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.559808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:36023 (local address 127.0.0.1:36222)
0807 22:45:48.560083 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.560092 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.560161 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.560350 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.560351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.560640 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.560704 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.560706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.561937 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.561938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.563489 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.563494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.564272 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.564273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.564373 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.589911 (+ 25538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.589917 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.591848 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.595254 (+ 3406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.595255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.595256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.595509 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.595858 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.595859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.595860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.596020 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.596604 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.596605 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.596661 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.603098 (+ 6437us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.603259 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":55,"threads_started":1}
I20260807 22:45:48.606371 23851 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221: attempting to promote NON_VOTER 9b7eb91351014fcd878bb334667f3c43 to VOTER
I20260807 22:45:48.606741 23851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } }
I20260807 22:45:48.607777 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.607370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:35661 (local address 127.0.0.1:56232)
0807 22:45:48.607659 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.607730 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264}
I20260807 22:45:48.608253 23710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 LEARNER]: Refusing update from remote peer 059a0a6fe8314dc4a77f3e93df688221: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260807 22:45:48.608418 23660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.608337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:35661 (local address 127.0.0.1:56236)
0807 22:45:48.608368 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.608392 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260807 22:45:48.608484 23075 consensus_peers.cc:487] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 -> Peer 0cb3166b76864a979588c5abf24a5ba8 (127.16.98.188:35661): Couldn't send request to peer 0cb3166b76864a979588c5abf24a5ba8. Status: Network error: Client connection negotiation failed: client connection to 127.16.98.188:35661: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 22:45:48.608744 23662 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 22:45:48.609722 23662 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9b7eb91351014fcd878bb334667f3c43 (127.16.98.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } } }
I20260807 22:45:48.610883 23662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } } }
I20260807 22:45:48.610965 23662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:48.611037 23851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } } }
I20260807 22:45:48.611272 23851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:48.611460 23710 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9b7eb91351014fcd878bb334667f3c43 (127.16.98.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } } }
I20260807 22:45:48.612928 23754 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.001s sys 0.000s
I20260807 22:45:48.613075 23855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } } }
I20260807 22:45:48.613269 23855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:45:48.613581 23856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:48.613709 23754 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:48.613760 23754 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260807 22:45:48.613783 23754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/wal
uuid: "0cb3166b76864a979588c5abf24a5ba8"
format_stamp: "Formatted at 2026-08-07 22:45:48 on dist-test-slave-rgb1"
server_key: "afb11465cdb4ca64604abbd908837025"
server_key_iv: "9ee4e3c312de0634612ab8e29d5fff71"
server_key_version: "encryptionkey@0"
I20260807 22:45:48.614740 23754 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:48.616665 23754 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 22:45:48.616768 23754 log.cc:1196] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260807 22:45:48.617018 23754 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: Deleting consensus metadata
I20260807 22:45:48.617154 23754 master_runner.cc:315] Copying system tablet from 127.16.98.190:35275
I20260807 22:45:48.617888 23754 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.98.190:35275
I20260807 22:45:48.626109 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.618322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.16.98.190:35275)
0807 22:45:48.618390 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.618393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.618451 (+ 58us) server_negotiation.cc:407] Connection header received
0807 22:45:48.618533 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.618534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.618548 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.618637 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.619066 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.619909 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.621154 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.621479 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.621560 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.622859 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.622860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.622861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.622869 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.624752 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.625208 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.625209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.625210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.625278 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.625509 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.625510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.625510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.625663 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.625738 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.625919 (+ 181us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.626008 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 22:45:48.626137 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.618224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:39008)
0807 22:45:48.618388 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.618400 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.618465 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.618647 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.618648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.618890 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.618973 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.618975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.620020 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.620022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.621053 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.621056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.621572 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.621573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.621649 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.622277 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.622283 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.622755 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.624848 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.624850 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.624850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.625119 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.625363 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.625364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.625364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.625435 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.625740 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.625741 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.625794 (+ 53us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.625875 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.626002 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260807 22:45:48.626925 23111 tablet_copy_service.cc:140] P 059a0a6fe8314dc4a77f3e93df688221: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0cb3166b76864a979588c5abf24a5ba8 ({username='oryx', principal='oryx/127.16.98.188@KRBTEST.COM'} at 127.0.0.1:39008)
I20260807 22:45:48.626998 23111 tablet_copy_service.cc:161] P 059a0a6fe8314dc4a77f3e93df688221: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0cb3166b76864a979588c5abf24a5ba8 at {username='oryx', principal='oryx/127.16.98.188@KRBTEST.COM'} at 127.0.0.1:39008: session id = 0cb3166b76864a979588c5abf24a5ba8-00000000000000000000000000000000
I20260807 22:45:48.627692 23111 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 22:45:48.628346 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 22:45:48.629925 23754 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: tablet copy: Starting download of 0 data blocks...
I20260807 22:45:48.630106 23754 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: tablet copy: Starting download of 1 WAL segments...
I20260807 22:45:48.631049 23754 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 22:45:48.632112 23111 tablet_copy_service.cc:342] P 059a0a6fe8314dc4a77f3e93df688221: Request end of tablet copy session 0cb3166b76864a979588c5abf24a5ba8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.98.188@KRBTEST.COM'} at 127.0.0.1:39008
I20260807 22:45:48.632180 23111 tablet_copy_service.cc:433] P 059a0a6fe8314dc4a77f3e93df688221: ending tablet copy session 0cb3166b76864a979588c5abf24a5ba8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0cb3166b76864a979588c5abf24a5ba8
I20260807 22:45:48.635579 23754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:48.637113 23863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:48.637574 23864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:48.638015 23754 server_base.cc:1034] running on GCE node
W20260807 22:45:48.638039 23866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:48.638461 23754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:48.638640 23754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:48.639780 23754 hybrid_clock.cc:648] HybridClock initialized: now 1786142748639772 us; error 28 us; skew 500 ppm
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:48.641489 23754 init.cc:377] Logged in from keytab as oryx/127.16.98.188@KRBTEST.COM (short username oryx)
I20260807 22:45:48.641786 23754 webserver.cc:460] Webserver started at http://127.16.98.188:35689/ using document root <none> and password file <none>
I20260807 22:45:48.641912 23754 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:48.641950 23754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:48.642722 23754 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:48.643335 23873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:48.643448 23754 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:48.643486 23754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/wal
uuid: "0cb3166b76864a979588c5abf24a5ba8"
format_stamp: "Formatted at 2026-08-07 22:45:48 on dist-test-slave-rgb1"
server_key: "afb11465cdb4ca64604abbd908837025"
server_key_iv: "9ee4e3c312de0634612ab8e29d5fff71"
server_key_version: "encryptionkey@0"
I20260807 22:45:48.643549 23754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:48.652904 23754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:48.654873 23754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:48.658315 23754 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.188:35661
I20260807 22:45:48.658769 23754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/master-2/data/info.pb
I20260807 22:45:48.660719 23926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.188:35661 every 8 connection(s)
I20260807 22:45:48.661670 23927 sys_catalog.cc:263] Verifying existing consensus state
I20260807 22:45:48.662200 23927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: Bootstrap starting.
I20260807 22:45:48.665720 23927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:45:48.666107 23927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: Bootstrap complete.
I20260807 22:45:48.666412 23927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } }
I20260807 22:45:48.666517 23927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0cb3166b76864a979588c5abf24a5ba8, State: Initialized, Role: LEARNER
I20260807 22:45:48.666559 23927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } }
I20260807 22:45:48.666950 23927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:48.668308 23930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } } }
I20260807 22:45:48.668423 23930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [sys.catalog]: This master's current role is: LEARNER
I20260807 22:45:48.669073 23941 catalog_manager.cc:1259] Loaded cluster ID: 9d54760e43384d9bbd11c0ae1393fd44
I20260807 22:45:48.669152 23941 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: loading cluster ID for follower catalog manager: success
I20260807 22:45:48.670310 23941 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: acquiring CA information for follower catalog manager: success
I20260807 22:45:48.670615 23941 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 22:45:48.702298 23850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:48.702529 23850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:48.702559 23850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:48.703889 23850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:48.703967 23850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:48.704017 23850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:45:48.705886 23850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.129.keytab
--principal=oryx/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:42247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=42059
--tserver_master_addrs=127.16.98.190:35275,127.16.98.189:36023,127.16.98.188:35661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:48.706168 23850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:48.706481 23850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:48.713961 23947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:48.713920 23946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:48.714205 23850 server_base.cc:1034] running on GCE node
W20260807 22:45:48.714433 23950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:48.714785 23850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:48.715103 23850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:48.716323 23850 hybrid_clock.cc:648] HybridClock initialized: now 1786142748716296 us; error 45 us; skew 500 ppm
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:48.719436 23850 init.cc:377] Logged in from keytab as oryx/127.16.98.129@KRBTEST.COM (short username oryx)
I20260807 22:45:48.720082 23850 webserver.cc:460] Webserver started at http://127.16.98.129:42059/ using document root <none> and password file <none>
I20260807 22:45:48.720351 23850 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:48.720474 23850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:48.722149 23850 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:48.722877 23957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:48.723099 23850 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:48.723199 23850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "57e9c38c98ed4690a9a434fa701ceb3a"
format_stamp: "Formatted at 2026-08-07 22:45:47 on dist-test-slave-rgb1"
server_key: "1c3dfeb335884c82c1b85ab0cd7b2e0e"
server_key_iv: "1be2d0af34e72488a539cb5f68a69dda"
server_key_version: "encryptionkey@0"
I20260807 22:45:48.723601 23850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:48.734190 23850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:48.736789 23850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:48.736949 23850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:48.737340 23850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:48.737762 23850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:48.737799 23850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:48.737823 23850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:48.737855 23850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:48.745798 23850 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:42247
I20260807 22:45:48.745867 24071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:42247 every 8 connection(s)
I20260807 22:45:48.746258 23850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
I20260807 22:45:48.755093 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 23850
I20260807 22:45:48.755357 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 23289
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.188@KRBTEST.COM
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.189@KRBTEST.COM
I20260807 22:45:48.759740 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.747575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:55351 (local address 127.16.98.190:35275)
0807 22:45:48.747638 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.747640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.747670 (+ 30us) server_negotiation.cc:407] Connection header received
0807 22:45:48.748211 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.748212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.748224 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.748289 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.749136 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.749917 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.751353 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.751659 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.751738 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.755821 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.755823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.755823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.755833 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.757783 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.758359 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.758361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.758362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.758426 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.758738 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.758739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.758740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.758896 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.758944 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.759543 (+ 599us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.759634 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 22:45:48.760401 24078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.747444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.129:55351)
0807 22:45:48.747596 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.747608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.748119 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.748365 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.748369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.748814 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.749041 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.749046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.750017 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.750018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.751107 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.751110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.751835 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.751839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.751957 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.753034 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.753040 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.755661 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.757890 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.757900 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.757903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.758261 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.758512 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.758513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.758513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.758653 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.758998 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.759006 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.759337 (+ 331us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.759511 (+ 174us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.759710 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":283,"thread_start_us":84,"threads_started":1}
I20260807 22:45:48.761595 24073 heartbeater.cc:347] Connected to a master server at 127.16.98.190:35275
I20260807 22:45:48.761713 24073 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:48.761962 24073 heartbeater.cc:511] Master 127.16.98.190:35275 requested a full tablet report, sending...
I20260807 22:45:48.762549 23091 ts_manager.cc:194] Re-registered known tserver with Master: 57e9c38c98ed4690a9a434fa701ceb3a (127.16.98.129:42247)
I20260807 22:45:48.762610 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.747838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:50921 (local address 127.16.98.189:36023)
0807 22:45:48.747911 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.747919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.747930 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:45:48.748228 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.748229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.748242 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.748275 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.749174 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.749913 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.751381 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.751681 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.751740 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.759381 (+ 7641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.759383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.759383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.759394 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.761274 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.761729 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.761730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.761731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.761792 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.761998 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.762000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.762000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.762181 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.762264 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.762439 (+ 175us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.762518 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 22:45:48.762624 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.747640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:36023 (local address 127.16.98.129:50921)
0807 22:45:48.747757 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.747764 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.748118 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.748366 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.748370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.748905 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.749042 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.749049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.749996 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.749997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.751246 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.751250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.751826 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.751827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.751941 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.752960 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.752964 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.759249 (+ 6285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.761380 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.761381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.761381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.761631 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.761870 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.761871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.761871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.761928 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.762265 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.762265 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.762303 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.762397 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.762520 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":51,"thread_start_us":76,"threads_started":1}
I20260807 22:45:48.763224 23091 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:55351
I20260807 22:45:48.763903 24074 heartbeater.cc:347] Connected to a master server at 127.16.98.189:36023
I20260807 22:45:48.763957 24074 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:48.764071 24074 heartbeater.cc:511] Master 127.16.98.189:36023 requested a full tablet report, sending...
I20260807 22:45:48.764443 23700 ts_manager.cc:194] Registered new tserver with Master: 57e9c38c98ed4690a9a434fa701ceb3a (127.16.98.129:42247)
I20260807 22:45:48.765009 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:45055
--local_ip_for_outbound_sockets=127.16.98.130
--tserver_master_addrs=127.16.98.190:35275
--webserver_port=40667
--webserver_interface=127.16.98.130
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.130.keytab
--principal=oryx/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.98.190:35275,127.16.98.189:36023
--tserver_master_addrs=127.16.98.190:35275,127.16.98.189:36023,127.16.98.188:35661 with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:45:48.768132 24076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.747117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:49289 (local address 127.16.98.188:35661)
0807 22:45:48.747241 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.747244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.747451 (+ 207us) server_negotiation.cc:407] Connection header received
0807 22:45:48.748272 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.748273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.748284 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.748343 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.749159 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.749804 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.751368 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.751853 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.751999 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.765030 (+ 13031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.765033 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.765033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.765045 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.766884 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.767263 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.767264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.767264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.767333 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.767540 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.767541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.767542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.767675 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.767750 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.767934 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.768027 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":82,"threads_started":1}
I20260807 22:45:48.768349 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.747105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:35661 (local address 127.16.98.129:49289)
0807 22:45:48.747287 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.747333 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.748102 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.748388 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.748389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.748818 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.749042 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.749049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.749929 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.749930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.751276 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.751278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.752009 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.752010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.752078 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.752937 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.752949 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.764915 (+ 11966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.766986 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.766987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.766987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.767203 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.767411 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.767412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.767412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.767477 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.767754 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.767755 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.767801 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.767890 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.768025 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":368,"thread_start_us":103,"threads_started":1}
I20260807 22:45:48.768829 24072 heartbeater.cc:347] Connected to a master server at 127.16.98.188:35661
I20260807 22:45:48.768887 24072 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:48.768976 24072 heartbeater.cc:511] Master 127.16.98.188:35661 requested a full tablet report, sending...
I20260807 22:45:48.769380 23890 ts_manager.cc:194] Registered new tserver with Master: 57e9c38c98ed4690a9a434fa701ceb3a (127.16.98.129:42247)
W20260807 22:45:48.853937 24080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:48.854226 24080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:48.854269 24080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:48.855504 24080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:48.855583 24080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:48.855633 24080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:45:48.857290 24080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.130.keytab
--principal=oryx/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:45055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=40667
--tserver_master_addrs=127.16.98.190:35275,127.16.98.189:36023,127.16.98.188:35661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:48.857607 24080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:48.857863 24080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 22:45:48.865376 24080 server_base.cc:1034] running on GCE node
W20260807 22:45:48.865334 24088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:48.865334 24086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:48.866284 24085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:48.866645 24080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:48.867025 24080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:48.868201 24080 hybrid_clock.cc:648] HybridClock initialized: now 1786142748868172 us; error 46 us; skew 500 ppm
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:48.872161 24080 init.cc:377] Logged in from keytab as oryx/127.16.98.130@KRBTEST.COM (short username oryx)
I20260807 22:45:48.872807 24080 webserver.cc:460] Webserver started at http://127.16.98.130:40667/ using document root <none> and password file <none>
I20260807 22:45:48.873077 24080 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:48.873167 24080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:48.875077 24080 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:48.876367 24096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:48.877008 24080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260807 22:45:48.877207 24080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a"
format_stamp: "Formatted at 2026-08-07 22:45:47 on dist-test-slave-rgb1"
server_key: "c5a4b7b76a003206b7ff0a2c1eb55929"
server_key_iv: "e79aadff6546969ce361ba226563a4f6"
server_key_version: "encryptionkey@0"
I20260807 22:45:48.877790 24080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:48.928772 24080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:48.931559 24080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:48.931769 24080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:48.932140 24080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:48.932611 24080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:48.932657 24080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:48.932700 24080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:48.932727 24080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:48.939428 24080 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:45055
I20260807 22:45:48.939478 24209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:45055 every 8 connection(s)
I20260807 22:45:48.939877 24080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:45:48.943137 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 24080
I20260807 22:45:48.943301 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 23424
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.188@KRBTEST.COM
Aug 07 22:45:48 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142748, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.189@KRBTEST.COM
I20260807 22:45:48.952602 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:43797
--local_ip_for_outbound_sockets=127.16.98.131
--tserver_master_addrs=127.16.98.190:35275
--webserver_port=43989
--webserver_interface=127.16.98.131
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.131.keytab
--principal=oryx/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.98.190:35275,127.16.98.189:36023
--tserver_master_addrs=127.16.98.190:35275,127.16.98.189:36023,127.16.98.188:35661 with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:45:48.954456 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.941198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:55017 (local address 127.16.98.190:35275)
0807 22:45:48.941245 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.941247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.941293 (+ 46us) server_negotiation.cc:407] Connection header received
0807 22:45:48.941808 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.941809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.941822 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.941855 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.942741 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.943524 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.945546 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.945931 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.945993 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.949429 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.949431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.949431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.949440 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.951580 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.953296 (+ 1716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.953298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.953298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.953369 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.953632 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.953633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.953633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.953786 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.953847 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.954254 (+ 407us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.954353 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 22:45:48.954496 24076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.940718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:56935 (local address 127.16.98.188:35661)
0807 22:45:48.940840 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.940843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.941150 (+ 307us) server_negotiation.cc:407] Connection header received
0807 22:45:48.941808 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.941810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.941825 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.941891 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.942781 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.943635 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.947711 (+ 4076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.948027 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.948070 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.950806 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.950808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.950808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.950817 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.952645 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.953229 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.953231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.953232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.953294 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.953611 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.953612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.953613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.953767 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.953812 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.954250 (+ 438us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.954370 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 22:45:48.954586 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.940896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:51581 (local address 127.16.98.189:36023)
0807 22:45:48.941014 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:48.941017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:48.941154 (+ 137us) server_negotiation.cc:407] Connection header received
0807 22:45:48.941809 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:48.941810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:48.941822 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:48.941855 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:48.942742 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.943611 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.947772 (+ 4161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.948080 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.948121 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.951643 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:48.951645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:48.951645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:48.951655 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:48.953358 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.953755 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.953756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.953757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.953814 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:48.954034 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:48.954035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:48.954035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:48.954166 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:48.954213 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:48.954419 (+ 206us) server_negotiation.cc:299] Negotiation successful
0807 22:45:48.954496 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 22:45:48.955073 24215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.940741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:36023 (local address 127.16.98.130:51581)
0807 22:45:48.941069 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.941078 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.941705 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.942360 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.942360 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.942529 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.942626 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.942632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.943727 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.943728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.947672 (+ 3944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.947675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.948194 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.948195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.948251 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.948981 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.948998 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.951521 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.953446 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.953447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.953447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.953672 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.953889 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.953890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.953890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.953946 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.954266 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.954266 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.954300 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.954694 (+ 394us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.954895 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":2930,"thread_start_us":108,"threads_started":1}
I20260807 22:45:48.955317 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.941105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.130:55017)
0807 22:45:48.941240 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.941251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.941705 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.942067 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.942068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.942362 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.942626 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.942632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.943678 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.943679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.945428 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.945432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.946088 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.946089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.946266 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.946910 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.946923 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.949311 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.951726 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.951743 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.951745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.953190 (+ 1445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.953446 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.953447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.953447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.953548 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.953880 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.953882 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.954080 (+ 198us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.954898 (+ 818us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.955024 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":89,"thread_start_us":84,"threads_started":1}
I20260807 22:45:48.955560 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:48.940742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:35661 (local address 127.16.98.130:56935)
0807 22:45:48.941008 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:48.941058 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:48.941682 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:48.941986 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:48.941990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:48.942355 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:48.942625 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.942630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.943728 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.943729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.947575 (+ 3846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:48.947579 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:48.948162 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:48.948163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:48.948229 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:48.949127 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:48.949132 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:48.950693 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:48.952721 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.952722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.952722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.953128 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.953370 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:48.953371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:48.953371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:48.953531 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:48.953860 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:48.953867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:48.954080 (+ 213us) client_negotiation.cc:769] Sending connection context
0807 22:45:48.954662 (+ 582us) client_negotiation.cc:240] Negotiation successful
0807 22:45:48.955437 (+ 775us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":2792,"thread_start_us":110,"threads_started":1}
I20260807 22:45:48.955801 24212 heartbeater.cc:347] Connected to a master server at 127.16.98.189:36023
I20260807 22:45:48.955899 24212 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:48.955896 24210 heartbeater.cc:347] Connected to a master server at 127.16.98.188:35661
I20260807 22:45:48.955963 24210 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:48.956139 24212 heartbeater.cc:511] Master 127.16.98.189:36023 requested a full tablet report, sending...
I20260807 22:45:48.956139 24210 heartbeater.cc:511] Master 127.16.98.188:35661 requested a full tablet report, sending...
I20260807 22:45:48.956382 24211 heartbeater.cc:347] Connected to a master server at 127.16.98.190:35275
I20260807 22:45:48.956420 24211 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:48.956477 24211 heartbeater.cc:511] Master 127.16.98.190:35275 requested a full tablet report, sending...
I20260807 22:45:48.956743 23890 ts_manager.cc:194] Registered new tserver with Master: f89601ecd4ae42c59e1ffbd5bd25ad1a (127.16.98.130:45055)
I20260807 22:45:48.956743 23700 ts_manager.cc:194] Registered new tserver with Master: f89601ecd4ae42c59e1ffbd5bd25ad1a (127.16.98.130:45055)
I20260807 22:45:48.957165 23091 ts_manager.cc:194] Re-registered known tserver with Master: f89601ecd4ae42c59e1ffbd5bd25ad1a (127.16.98.130:45055)
I20260807 22:45:48.957706 23091 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:55017
W20260807 22:45:49.042459 24217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:49.042687 24217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:49.042714 24217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:49.043937 24217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:49.043992 24217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:49.044020 24217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:45:49.045737 24217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.131.keytab
--principal=oryx/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:43797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=43989
--tserver_master_addrs=127.16.98.190:35275,127.16.98.189:36023,127.16.98.188:35661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:49.045957 24217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:49.046159 24217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:49.051661 24222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:49.051810 24217 server_base.cc:1034] running on GCE node
W20260807 22:45:49.051681 24223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:49.051939 24225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:49.052155 24217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:49.052440 24217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:49.053619 24217 hybrid_clock.cc:648] HybridClock initialized: now 1786142749053583 us; error 54 us; skew 500 ppm
Aug 07 22:45:49 dist-test-slave-rgb1 krb5kdc[23045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142749, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:49.056679 24217 init.cc:377] Logged in from keytab as oryx/127.16.98.131@KRBTEST.COM (short username oryx)
I20260807 22:45:49.057228 24217 webserver.cc:460] Webserver started at http://127.16.98.131:43989/ using document root <none> and password file <none>
I20260807 22:45:49.057508 24217 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:49.057603 24217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:49.059201 24217 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:45:49.059958 24232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:49.060382 24217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:49.060523 24217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "93be69dbc9034b229bbc074bc8fde968"
format_stamp: "Formatted at 2026-08-07 22:45:47 on dist-test-slave-rgb1"
server_key: "e522c015eb28ac0ebc3d49e56c5e8708"
server_key_iv: "1e61a540494cb961d6f91a06c1bede21"
server_key_version: "encryptionkey@0"
I20260807 22:45:49.061151 24217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:49.088430 24217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:49.091176 24217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:49.091403 24217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:49.091714 24217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:49.092075 24217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:49.092136 24217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:49.092185 24217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:49.092232 24217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:49.099198 24217 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:43797
I20260807 22:45:49.099253 24346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:43797 every 8 connection(s)
I20260807 22:45:49.099684 24217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:45:49.101946 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 24217
Aug 07 22:45:49 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142749, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.188@KRBTEST.COM
Aug 07 22:45:49 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142749, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
Aug 07 22:45:49 dist-test-slave-rgb1 krb5kdc[23045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142749, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.189@KRBTEST.COM
I20260807 22:45:49.115190 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.100621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:35661 (local address 127.16.98.131:60187)
0807 22:45:49.100807 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:49.100850 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:49.102018 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:49.102408 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:49.102412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:49.102910 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:49.103221 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.103227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.104656 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.104657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.105667 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.105669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.106194 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.106195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.106339 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.107050 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:49.107059 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:49.110621 (+ 3562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:49.113046 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:49.113057 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:49.113060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:49.113379 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:49.113654 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:49.113655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:49.113655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:49.113825 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:49.114326 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:49.114335 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:49.114542 (+ 207us) client_negotiation.cc:769] Sending connection context
0807 22:45:49.114661 (+ 119us) client_negotiation.cc:240] Negotiation successful
0807 22:45:49.114879 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":5,"thread_start_us":102,"threads_started":1}
I20260807 22:45:49.115612 24076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.100592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:60187 (local address 127.16.98.188:35661)
0807 22:45:49.100644 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.100647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.100933 (+ 286us) server_negotiation.cc:407] Connection header received
0807 22:45:49.102160 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.102161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.102176 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.102212 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:49.103352 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.104230 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.105752 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.106072 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.106130 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.110700 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:49.110702 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:49.110702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:49.110711 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:49.112953 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.113495 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.113497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.113497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.113569 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.114051 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.114052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.114052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.114224 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:49.114293 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.115386 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.115488 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 22:45:49.116254 24347 heartbeater.cc:347] Connected to a master server at 127.16.98.188:35661
I20260807 22:45:49.116369 24347 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:49.116674 24347 heartbeater.cc:511] Master 127.16.98.188:35661 requested a full tablet report, sending...
I20260807 22:45:49.117087 24352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.100622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.131:51135)
0807 22:45:49.100823 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:49.100851 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:49.102018 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:49.102409 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:49.102412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:49.102909 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:49.103221 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.103227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.104053 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.104054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.106274 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.106276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.106745 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.106747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.106811 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.107725 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:49.107731 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:49.110476 (+ 2745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:49.115589 (+ 5113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:49.115590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:49.115590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:49.115853 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:49.116095 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:49.116097 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:49.116097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:49.116166 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:49.116691 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:49.116693 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:49.116746 (+ 53us) client_negotiation.cc:769] Sending connection context
0807 22:45:49.116811 (+ 65us) client_negotiation.cc:240] Negotiation successful
0807 22:45:49.116955 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":4,"thread_start_us":132,"threads_started":1}
I20260807 22:45:49.117161 23890 ts_manager.cc:194] Registered new tserver with Master: 93be69dbc9034b229bbc074bc8fde968 (127.16.98.131:43797)
I20260807 22:45:49.117417 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.100628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:51135 (local address 127.16.98.190:35275)
0807 22:45:49.100695 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.100697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.100945 (+ 248us) server_negotiation.cc:407] Connection header received
0807 22:45:49.102161 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.102162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.102176 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.102212 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:49.103336 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.103909 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.106367 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.106660 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.106745 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.110716 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:49.110717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:49.110717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:49.110724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:49.114138 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.115943 (+ 1805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.115945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.115945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.116011 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.116242 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.116243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.116244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.116527 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:49.116656 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.117226 (+ 570us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.117317 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 22:45:49.117924 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.104202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.16.98.190:35275)
0807 22:45:49.104901 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.104903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.104912 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:45:49.104932 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.104932 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.104946 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.105027 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:49.105962 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.106774 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.108025 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.108362 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.108416 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.110233 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:49.110235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:49.110235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:49.110243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:49.115491 (+ 5248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.116848 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.116849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.116849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.116916 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.117151 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.117151 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.117152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.117291 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:49.117383 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.117696 (+ 313us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.117790 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":75,"threads_started":1}
I20260807 22:45:49.118420 24348 heartbeater.cc:347] Connected to a master server at 127.16.98.190:35275
I20260807 22:45:49.118465 24348 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:49.118542 24348 heartbeater.cc:511] Master 127.16.98.190:35275 requested a full tablet report, sending...
I20260807 22:45:49.119336 23090 ts_manager.cc:194] Re-registered known tserver with Master: 93be69dbc9034b229bbc074bc8fde968 (127.16.98.131:43797)
I20260807 22:45:49.119835 23090 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:51135
I20260807 22:45:49.120360 24362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.105221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.16.98.188:35661)
0807 22:45:49.105461 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.105463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.106887 (+ 1424us) server_negotiation.cc:407] Connection header received
0807 22:45:49.106992 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.106993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.107006 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.107045 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:49.107834 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.108777 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.109614 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.111032 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.111091 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.112595 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:49.112596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:49.112596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:49.112603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:49.118270 (+ 5667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.118804 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.118805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.118806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.118876 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.119297 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.119299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.119299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.119667 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:49.119765 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.120170 (+ 405us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.120263 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":69,"threads_started":1}
I20260807 22:45:49.121162 24360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.104745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.16.98.189:36023)
0807 22:45:49.109671 (+ 4926us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.109673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.109680 (+ 7us) server_negotiation.cc:407] Connection header received
0807 22:45:49.109700 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.109701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.109712 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.110360 (+ 648us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:49.110402 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.111151 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.112340 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.112687 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.112977 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.114196 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:49.114197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:49.114198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:49.114206 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:49.118971 (+ 4765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.119407 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.119408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.119409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.119474 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.119705 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.119706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.119707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.119839 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:49.120878 (+ 1039us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.120919 (+ 41us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.121043 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4904,"thread_start_us":71,"threads_started":1}
I20260807 22:45:49.121763 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.101167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:60317 (local address 127.16.98.189:36023)
0807 22:45:49.101224 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.101226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.101235 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:45:49.102525 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.102526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.102539 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.102600 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:49.103345 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.104230 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.105782 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.106113 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.106157 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.115820 (+ 9663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:49.115821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:49.115822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:49.115829 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:49.118159 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.120843 (+ 2684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.120844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.120844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.120910 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.121163 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.121164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.121165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.121343 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:49.121415 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.121635 (+ 220us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.121711 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 22:45:49.121860 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.100957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:36023 (local address 127.16.98.131:60317)
0807 22:45:49.101096 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:49.101108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:49.102298 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:49.102628 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:49.102630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:49.102922 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:49.103222 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.103230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.104354 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.104355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.105668 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.105671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.106407 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.106409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.106481 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.107563 (+ 1082us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:49.107570 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:49.115586 (+ 8016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:49.120033 (+ 4447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:49.120034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:49.120034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:49.120368 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:49.120993 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:49.120993 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:49.120994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:49.121058 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:49.121448 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:49.121449 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:49.121496 (+ 47us) client_negotiation.cc:769] Sending connection context
0807 22:45:49.121591 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 22:45:49.121727 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":34,"thread_start_us":71,"threads_started":1}
I20260807 22:45:49.122206 24349 heartbeater.cc:347] Connected to a master server at 127.16.98.189:36023
I20260807 22:45:49.122268 24349 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:49.119848 23091 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 22:45:49.122352 24349 heartbeater.cc:511] Master 127.16.98.189:36023 requested a full tablet report, sending...
W20260807 22:45:49.122587 23091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:49.122929 23700 ts_manager.cc:194] Registered new tserver with Master: 93be69dbc9034b229bbc074bc8fde968 (127.16.98.131:43797)
I20260807 22:45:49.131336 24375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.127497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:42247 (local address 127.0.0.1:60830)
0807 22:45:49.127602 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:49.127612 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:49.127685 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:49.128280 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:49.128280 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:49.128283 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:49.128333 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.128335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.129327 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.129328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.130378 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.130381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.131099 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.131100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.131160 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.131164 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:49.131212 (+ 48us) client_negotiation.cc:240] Negotiation successful
0807 22:45:49.131254 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":29,"thread_start_us":71,"threads_started":1}
I20260807 22:45:49.131345 24377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.127921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.16.98.129:42247)
0807 22:45:49.128101 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.128105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.128116 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:45:49.128158 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.128160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.128176 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.128407 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:49.128468 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.129227 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.130469 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.131008 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.131098 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.131190 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.131224 (+ 34us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.131265 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260807 22:45:49.131606 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.127248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:43797 (local address 127.0.0.1:34910)
0807 22:45:49.127387 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:49.127400 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:49.127458 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:49.127802 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:49.127803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:49.127805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:49.127862 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.127864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.129174 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.129175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.130368 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.130371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.131335 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.131336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.131400 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.131405 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:45:49.131438 (+ 33us) client_negotiation.cc:240] Negotiation successful
0807 22:45:49.131489 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":93,"threads_started":1}
I20260807 22:45:49.131845 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.127285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:45055 (local address 127.0.0.1:55338)
0807 22:45:49.127423 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:49.127435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:49.127490 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:49.128133 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:49.128134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:49.128137 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:49.128180 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.128182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.129466 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.129467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.130568 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.130570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.131639 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.131640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.131709 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.131713 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:49.131743 (+ 30us) client_negotiation.cc:240] Negotiation successful
0807 22:45:49.131784 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":95,"threads_started":1}
I20260807 22:45:49.132210 24374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.127335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34910 (local address 127.16.98.131:43797)
0807 22:45:49.127649 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.127655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.127665 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:45:49.127705 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.127707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.127726 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.127800 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:49.127927 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.129074 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.130499 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.131213 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.131773 (+ 560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.131847 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.131873 (+ 26us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.131919 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":87,"threads_started":1}
I20260807 22:45:49.132143 24006 tablet_service.cc:1467] Processing CreateTablet for tablet 397a4ad2b2f94271ab0affed3b7dc49f (DEFAULT_TABLE table=test-table [id=c52673805a86434bbf7551b7d76bec4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:49.132503 24376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.127726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55338 (local address 127.16.98.130:45055)
0807 22:45:49.127927 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.127936 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.127947 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:45:49.127988 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.127990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.128006 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.128256 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:49.128310 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.129375 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.130692 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.131382 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.132235 (+ 853us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.132329 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.132358 (+ 29us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.132404 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":95,"threads_started":1}
I20260807 22:45:49.132616 24006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397a4ad2b2f94271ab0affed3b7dc49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:49.132810 24280 tablet_service.cc:1467] Processing CreateTablet for tablet 397a4ad2b2f94271ab0affed3b7dc49f (DEFAULT_TABLE table=test-table [id=c52673805a86434bbf7551b7d76bec4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:49.133055 24280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397a4ad2b2f94271ab0affed3b7dc49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:49.133139 24144 tablet_service.cc:1467] Processing CreateTablet for tablet 397a4ad2b2f94271ab0affed3b7dc49f (DEFAULT_TABLE table=test-table [id=c52673805a86434bbf7551b7d76bec4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:49.133458 24144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397a4ad2b2f94271ab0affed3b7dc49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:49.135581 24378 tablet_bootstrap.cc:492] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968: Bootstrap starting.
I20260807 22:45:49.136238 24378 tablet_bootstrap.cc:654] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:49.136323 24379 tablet_bootstrap.cc:492] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a: Bootstrap starting.
I20260807 22:45:49.136634 24378 log.cc:824] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:49.136978 24379 tablet_bootstrap.cc:654] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:49.137306 24379 log.cc:824] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:49.137470 24378 tablet_bootstrap.cc:492] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968: No bootstrap required, opened a new log
I20260807 22:45:49.137557 24378 ts_tablet_manager.cc:1397] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 22:45:49.137641 24381 tablet_bootstrap.cc:492] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a: Bootstrap starting.
I20260807 22:45:49.138180 24381 tablet_bootstrap.cc:654] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:49.138481 24381 log.cc:824] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:49.138996 24378 raft_consensus.cc:348] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } }
I20260807 22:45:49.139200 24378 raft_consensus.cc:374] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:49.139241 24378 raft_consensus.cc:729] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93be69dbc9034b229bbc074bc8fde968, State: Initialized, Role: FOLLOWER
I20260807 22:45:49.139338 24381 tablet_bootstrap.cc:492] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a: No bootstrap required, opened a new log
I20260807 22:45:49.139376 24379 tablet_bootstrap.cc:492] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a: No bootstrap required, opened a new log
I20260807 22:45:49.139406 24381 ts_tablet_manager.cc:1397] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 22:45:49.139370 24378 consensus_queue.cc:260] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } }
I20260807 22:45:49.139452 24379 ts_tablet_manager.cc:1397] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260807 22:45:49.139667 24378 ts_tablet_manager.cc:1428] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:45:49.139780 24348 heartbeater.cc:503] Master 127.16.98.190:35275 was elected leader, sending a full tablet report...
I20260807 22:45:49.141535 24381 raft_consensus.cc:348] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } }
I20260807 22:45:49.141724 24381 raft_consensus.cc:374] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:49.141776 24381 raft_consensus.cc:729] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f89601ecd4ae42c59e1ffbd5bd25ad1a, State: Initialized, Role: FOLLOWER
I20260807 22:45:49.141893 24381 consensus_queue.cc:260] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } }
I20260807 22:45:49.142210 24381 ts_tablet_manager.cc:1428] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:45:49.142228 24211 heartbeater.cc:503] Master 127.16.98.190:35275 was elected leader, sending a full tablet report...
I20260807 22:45:49.142292 24379 raft_consensus.cc:348] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } }
I20260807 22:45:49.142473 24379 raft_consensus.cc:374] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:49.142517 24379 raft_consensus.cc:729] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e9c38c98ed4690a9a434fa701ceb3a, State: Initialized, Role: FOLLOWER
I20260807 22:45:49.142637 24379 consensus_queue.cc:260] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } }
I20260807 22:45:49.142902 24379 ts_tablet_manager.cc:1428] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260807 22:45:49.143216 24073 heartbeater.cc:503] Master 127.16.98.190:35275 was elected leader, sending a full tablet report...
W20260807 22:45:49.191030 24213 tablet.cc:2369] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:45:49.196471 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.193328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:35661 (local address 127.0.0.1:56266)
0807 22:45:49.193411 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:49.193425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:49.193489 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:49.193800 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:49.193801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:49.193803 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:49.193871 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.193873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.194712 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.194713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.195599 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.195601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.196241 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.196242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.196300 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.196304 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:49.196363 (+ 59us) client_negotiation.cc:240] Negotiation successful
0807 22:45:49.196406 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 22:45:49.196502 24362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.193541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.16.98.188:35661)
0807 22:45:49.193613 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.193616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.193629 (+ 13us) server_negotiation.cc:407] Connection header received
0807 22:45:49.193667 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.193668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.193695 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.193782 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:49.193957 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.194607 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.195704 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.196145 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.196212 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.196268 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.196395 (+ 127us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.196444 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 22:45:49.197070 23900 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260807 22:45:49.198129 24387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 059a0a6fe8314dc4a77f3e93df688221. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } } }
I20260807 22:45:49.198249 24387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [sys.catalog]: This master's current role is: LEARNER
W20260807 22:45:49.247361 24075 tablet.cc:2369] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:45:49.269456 24385 raft_consensus.cc:482] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:49.269629 24385 raft_consensus.cc:504] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } }
I20260807 22:45:49.270079 24385 leader_election.cc:288] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93be69dbc9034b229bbc074bc8fde968 (127.16.98.131:43797), 57e9c38c98ed4690a9a434fa701ceb3a (127.16.98.129:42247)
I20260807 22:45:49.273736 24377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.270614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:60989 (local address 127.16.98.129:42247)
0807 22:45:49.270654 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.270656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.270672 (+ 16us) server_negotiation.cc:407] Connection header received
0807 22:45:49.270697 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.270698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.270706 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.270802 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:49.270978 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.271560 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.272842 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.273324 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.273386 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.273393 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.273616 (+ 223us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.273661 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260807 22:45:49.273742 24374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.270526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:51639 (local address 127.16.98.131:43797)
0807 22:45:49.270594 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.270597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.270694 (+ 97us) server_negotiation.cc:407] Connection header received
0807 22:45:49.270762 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.270763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.270781 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.270859 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:49.271087 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.271598 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.272769 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.273342 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.273455 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.273465 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.273636 (+ 171us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.273672 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 22:45:49.273759 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.270569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:43797 (local address 127.16.98.130:51639)
0807 22:45:49.270616 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:49.270628 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:49.270700 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:49.270863 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:49.270864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:49.270866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:49.270947 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.270949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.271743 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.271744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.272678 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.272681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.273463 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.273464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.273570 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.273573 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:45:49.273629 (+ 56us) client_negotiation.cc:240] Negotiation successful
0807 22:45:49.273673 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":19}
I20260807 22:45:49.273784 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.270386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:42247 (local address 127.16.98.130:60989)
0807 22:45:49.270460 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:49.270480 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:49.270536 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:49.270810 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:49.270812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:49.270814 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:49.270894 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.270896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.271677 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.271678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.272705 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:49.272708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.273414 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.273416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:49.273493 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.273498 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:45:49.273594 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 22:45:49.273638 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":56}
I20260807 22:45:49.274132 24026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397a4ad2b2f94271ab0affed3b7dc49f" candidate_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" is_pre_election: true
I20260807 22:45:49.274317 24026 raft_consensus.cc:2454] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f89601ecd4ae42c59e1ffbd5bd25ad1a in term 0.
I20260807 22:45:49.274302 24301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397a4ad2b2f94271ab0affed3b7dc49f" candidate_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93be69dbc9034b229bbc074bc8fde968" is_pre_election: true
I20260807 22:45:49.274441 24301 raft_consensus.cc:2454] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f89601ecd4ae42c59e1ffbd5bd25ad1a in term 0.
I20260807 22:45:49.274559 24097 leader_election.cc:302] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57e9c38c98ed4690a9a434fa701ceb3a, f89601ecd4ae42c59e1ffbd5bd25ad1a; no voters:
I20260807 22:45:49.274681 24385 raft_consensus.cc:2781] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:49.274735 24385 raft_consensus.cc:482] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:49.274758 24385 raft_consensus.cc:3037] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:49.275528 24385 raft_consensus.cc:504] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } }
I20260807 22:45:49.275672 24385 leader_election.cc:288] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [CANDIDATE]: Term 1 election: Requested vote from peers 93be69dbc9034b229bbc074bc8fde968 (127.16.98.131:43797), 57e9c38c98ed4690a9a434fa701ceb3a (127.16.98.129:42247)
I20260807 22:45:49.275818 24301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397a4ad2b2f94271ab0affed3b7dc49f" candidate_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93be69dbc9034b229bbc074bc8fde968"
I20260807 22:45:49.275847 24026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397a4ad2b2f94271ab0affed3b7dc49f" candidate_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e9c38c98ed4690a9a434fa701ceb3a"
I20260807 22:45:49.275899 24301 raft_consensus.cc:3037] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:49.275957 24026 raft_consensus.cc:3037] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:49.276800 24026 raft_consensus.cc:2454] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f89601ecd4ae42c59e1ffbd5bd25ad1a in term 1.
I20260807 22:45:49.276863 24301 raft_consensus.cc:2454] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f89601ecd4ae42c59e1ffbd5bd25ad1a in term 1.
I20260807 22:45:49.277074 24098 leader_election.cc:302] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93be69dbc9034b229bbc074bc8fde968, f89601ecd4ae42c59e1ffbd5bd25ad1a; no voters:
I20260807 22:45:49.277222 24385 raft_consensus.cc:2781] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:49.277407 24385 raft_consensus.cc:686] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 1 LEADER]: Becoming Leader. State: Replica: f89601ecd4ae42c59e1ffbd5bd25ad1a, State: Running, Role: LEADER
I20260807 22:45:49.277623 24385 consensus_queue.cc:237] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } }
I20260807 22:45:49.278628 23090 catalog_manager.cc:5495] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a reported cstate change: term changed from 0 to 1, leader changed from <none> to f89601ecd4ae42c59e1ffbd5bd25ad1a (127.16.98.130). New cstate: current_term: 1 leader_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } health_report { overall_health: UNKNOWN } } }
I20260807 22:45:49.279286 24366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: NON_VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: true } } }
I20260807 22:45:49.279392 24366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:49.279980 24366 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221: attempting to promote NON_VOTER 0cb3166b76864a979588c5abf24a5ba8 to VOTER
I20260807 22:45:49.280107 24366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } }
I20260807 22:45:49.280532 23710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 FOLLOWER]: Refusing update from remote peer 059a0a6fe8314dc4a77f3e93df688221: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260807 22:45:49.280546 23900 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 1 LEARNER]: Refusing update from remote peer 059a0a6fe8314dc4a77f3e93df688221: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260807 22:45:49.280793 24371 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260807 22:45:49.280928 24365 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260807 22:45:49.281494 24371 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 0cb3166b76864a979588c5abf24a5ba8 (127.16.98.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } } }
I20260807 22:45:49.281610 23710 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 0cb3166b76864a979588c5abf24a5ba8 (127.16.98.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } } }
I20260807 22:45:49.282496 24371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } } }
I20260807 22:45:49.282493 24366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } } }
I20260807 22:45:49.282612 24371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:49.282632 24366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:49.282667 24371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } } }
I20260807 22:45:49.282725 24371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:49.282819 23900 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 0cb3166b76864a979588c5abf24a5ba8 (127.16.98.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } } }
I20260807 22:45:49.283659 24387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } } }
I20260807 22:45:49.283744 24387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:45:49.284184 24394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "059a0a6fe8314dc4a77f3e93df688221" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } } }
I20260807 22:45:49.284271 24394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:45:49.287369 24376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.284939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.16.98.130:45055)
0807 22:45:49.285016 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.285017 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.285024 (+ 7us) server_negotiation.cc:407] Connection header received
0807 22:45:49.285052 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.285053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.285070 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.285149 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:49.285292 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.285781 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.286558 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.286754 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.286842 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.287026 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:49.287130 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:49.287221 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.287261 (+ 40us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.287286 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 22:45:49.289287 24301 raft_consensus.cc:1261] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968 [term 1 FOLLOWER]: Refusing update from remote peer f89601ecd4ae42c59e1ffbd5bd25ad1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:49.289300 24026 raft_consensus.cc:1261] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a [term 1 FOLLOWER]: Refusing update from remote peer f89601ecd4ae42c59e1ffbd5bd25ad1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:49.289646 24391 consensus_queue.cc:1035] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:49.289784 24385 consensus_queue.cc:1035] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:49.292332 24396 mvcc.cc:205] Tried to move back new op lower bound from 7316040701086081024 to 7316040701041807360. Current Snapshot: MvccSnapshot[applied={T|T < 7316040701086081024}]
I20260807 22:45:49.293715 24397 mvcc.cc:205] Tried to move back new op lower bound from 7316040701086081024 to 7316040701041807360. Current Snapshot: MvccSnapshot[applied={T|T < 7316040701086081024 or (T in {7316040701086081024})}]
I20260807 22:45:49.293850 24398 mvcc.cc:205] Tried to move back new op lower bound from 7316040701086081024 to 7316040701041807360. Current Snapshot: MvccSnapshot[applied={T|T < 7316040701086081024}]
I20260807 22:45:49.296164 23090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:49.296264 23090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:49.298249 23090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 059a0a6fe8314dc4a77f3e93df688221: Sending DeleteTablet for 3 replicas of tablet 397a4ad2b2f94271ab0affed3b7dc49f
I20260807 22:45:49.298648 24144 tablet_service.cc:1514] Processing DeleteTablet for tablet 397a4ad2b2f94271ab0affed3b7dc49f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:49 UTC) from {username='oryx', principal='oryx/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:55338
I20260807 22:45:49.298668 24280 tablet_service.cc:1514] Processing DeleteTablet for tablet 397a4ad2b2f94271ab0affed3b7dc49f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:49 UTC) from {username='oryx', principal='oryx/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:34910
I20260807 22:45:49.298864 24405 tablet_replica.cc:330] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a: stopping tablet replica
I20260807 22:45:49.298959 24405 raft_consensus.cc:2229] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:45:49.299081 24405 raft_consensus.cc:2258] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:49.299139 24006 tablet_service.cc:1514] Processing DeleteTablet for tablet 397a4ad2b2f94271ab0affed3b7dc49f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:49 UTC) from {username='oryx', principal='oryx/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:60830
I20260807 22:45:49.300009 24405 ts_tablet_manager.cc:1905] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:49.301172 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 23058
I20260807 22:45:49.303236 24405 ts_tablet_manager.cc:1918] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:49.303329 24405 log.cc:1196] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/397a4ad2b2f94271ab0affed3b7dc49f
I20260807 22:45:49.303637 24405 ts_tablet_manager.cc:1939] T 397a4ad2b2f94271ab0affed3b7dc49f P f89601ecd4ae42c59e1ffbd5bd25ad1a: Deleting consensus metadata
I20260807 22:45:49.304279 24407 tablet_replica.cc:330] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a: stopping tablet replica
I20260807 22:45:49.304461 24407 raft_consensus.cc:2229] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:49.304627 24407 raft_consensus.cc:2258] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:49.305657 24407 ts_tablet_manager.cc:1905] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:49.307161 24406 tablet_replica.cc:330] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968: stopping tablet replica
I20260807 22:45:49.307296 24406 raft_consensus.cc:2229] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:49.307418 24406 raft_consensus.cc:2258] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:49.307962 24406 ts_tablet_manager.cc:1905] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:49.310031 24406 ts_tablet_manager.cc:1918] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:49.310115 24406 log.cc:1196] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/397a4ad2b2f94271ab0affed3b7dc49f
W20260807 22:45:49.310621 24099 connection.cc:535] server connection from 127.0.0.1:55338 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 22:45:49.311599 24407 ts_tablet_manager.cc:1918] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:49.311690 24407 log.cc:1196] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-0/wal/wals/397a4ad2b2f94271ab0affed3b7dc49f
I20260807 22:45:49.312050 24407 ts_tablet_manager.cc:1939] T 397a4ad2b2f94271ab0affed3b7dc49f P 57e9c38c98ed4690a9a434fa701ceb3a: Deleting consensus metadata
W20260807 22:45:49.312513 23959 connection.cc:441] server connection from 127.0.0.1:60830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60830 (request call id 1) could send its response
I20260807 22:45:49.312862 24406 ts_tablet_manager.cc:1939] T 397a4ad2b2f94271ab0affed3b7dc49f P 93be69dbc9034b229bbc074bc8fde968: Deleting consensus metadata
W20260807 22:45:49.313303 24236 connection.cc:441] server connection from 127.0.0.1:34910 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34910 (request call id 1) could send its response
I20260807 22:45:49.313345 24362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.300979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.16.98.188:35661)
0807 22:45:49.301037 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.301039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.302900 (+ 1861us) server_negotiation.cc:407] Connection header received
0807 22:45:49.303007 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.303008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.303020 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.303053 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:49.305246 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.306038 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.307206 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.307558 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.307610 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.309570 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:49.309571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:49.309571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:49.309579 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:49.311632 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.312231 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.312232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.312233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.312296 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.312511 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.312513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.312513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.312684 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:49.312772 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.313127 (+ 355us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.313228 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 22:45:49.322631 24362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.314003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56284 (local address 127.16.98.188:35661)
0807 22:45:49.314046 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.314048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.314074 (+ 26us) server_negotiation.cc:407] Connection header received
0807 22:45:49.314141 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.314142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.314149 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.314197 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:49.314905 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.315470 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.316717 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.317061 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.317180 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.318773 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:49.318782 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:49.318783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:49.318792 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:49.321157 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.321586 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.321588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.321588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.321649 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.321969 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.321970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.321971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.322120 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:49.322222 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.322390 (+ 168us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.322491 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260807 22:45:49.330803 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:49.323186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.16.98.189:36023)
0807 22:45:49.323229 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:49.323231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:49.323352 (+ 121us) server_negotiation.cc:407] Connection header received
0807 22:45:49.323411 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:49.323411 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:49.323445 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:49.323493 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:49.324117 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.324893 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.325967 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:49.326156 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:49.326283 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:49.327775 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:49.327777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:49.327777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:49.327785 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:49.329407 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.329864 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.329865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.329865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.329930 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:49.330176 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:49.330177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:49.330177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:49.330308 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:49.330428 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:49.330582 (+ 154us) server_negotiation.cc:299] Negotiation successful
0807 22:45:49.330681 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260807 22:45:50.294552 24414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:50.294243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.129:33757)
0807 22:45:50.294402 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:50.294488 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
W20260807 22:45:50.294939 24073 heartbeater.cc:644] Failed to heartbeat to 127.16.98.190:35275 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
I20260807 22:45:50.294941 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:50.294492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.131:57847)
0807 22:45:50.294774 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:50.294881 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":65,"threads_started":1}
W20260807 22:45:50.295128 24348 heartbeater.cc:644] Failed to heartbeat to 127.16.98.190:35275 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
I20260807 22:45:50.295260 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:50.294719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.130:35901)
0807 22:45:50.295144 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:50.295214 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":72,"threads_started":1}
W20260807 22:45:50.295449 24211 heartbeater.cc:644] Failed to heartbeat to 127.16.98.190:35275 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
I20260807 22:45:50.798594 24417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 059a0a6fe8314dc4a77f3e93df688221)
I20260807 22:45:50.798746 24417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } }
I20260807 22:45:50.798940 24418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 059a0a6fe8314dc4a77f3e93df688221)
I20260807 22:45:50.799113 24417 leader_election.cc:288] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 059a0a6fe8314dc4a77f3e93df688221 (127.16.98.190:35275), 9b7eb91351014fcd878bb334667f3c43 (127.16.98.189:36023)
I20260807 22:45:50.799142 24418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } }
I20260807 22:45:50.799366 24418 leader_election.cc:288] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 059a0a6fe8314dc4a77f3e93df688221 (127.16.98.190:35275), 0cb3166b76864a979588c5abf24a5ba8 (127.16.98.188:35661)
I20260807 22:45:50.799602 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:50.799252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:39446)
0807 22:45:50.799481 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:50.799539 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":38,"threads_started":1}
W20260807 22:45:50.799744 23874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
I20260807 22:45:50.799794 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:50.799483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:39450)
0807 22:45:50.799712 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:50.799757 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
W20260807 22:45:50.799932 23684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
I20260807 22:45:50.800199 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:50.800107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:39462)
0807 22:45:50.800147 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:50.800172 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260807 22:45:50.800294 23874 leader_election.cc:334] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 059a0a6fe8314dc4a77f3e93df688221 (127.16.98.190:35275): Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
I20260807 22:45:50.800388 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:50.800289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:39472)
0807 22:45:50.800330 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:50.800362 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260807 22:45:50.800460 23684 leader_election.cc:334] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 059a0a6fe8314dc4a77f3e93df688221 (127.16.98.190:35275): Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
I20260807 22:45:50.802649 24422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:50.799675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56294 (local address 127.16.98.188:35661)
0807 22:45:50.799775 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:50.799777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:50.799995 (+ 218us) server_negotiation.cc:407] Connection header received
0807 22:45:50.800038 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:50.800039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:50.800049 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:50.800107 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:50.800296 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:50.800923 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:50.801870 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:50.802276 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:50.802365 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:50.802376 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:50.802540 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 22:45:50.802571 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":50,"threads_started":1}
I20260807 22:45:50.802654 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:50.799767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:35661 (local address 127.0.0.1:56294)
0807 22:45:50.799856 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:50.799867 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:50.799936 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:50.800157 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:50.800158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:50.800160 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:50.800219 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:50.800221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:50.801013 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:50.801014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:50.801777 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:50.801779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:50.802408 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:50.802409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:50.802466 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:50.802471 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:45:50.802525 (+ 54us) client_negotiation.cc:240] Negotiation successful
0807 22:45:50.802568 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":41,"threads_started":1}
I20260807 22:45:50.802850 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:50.799407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:36023 (local address 127.0.0.1:36256)
0807 22:45:50.799539 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:50.799551 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:50.799608 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:50.800368 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:50.800369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:50.800371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:50.800425 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:50.800444 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:50.801166 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:50.801167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:50.801974 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:50.801976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:50.802618 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:50.802619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:50.802682 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:50.802686 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:50.802745 (+ 59us) client_negotiation.cc:240] Negotiation successful
0807 22:45:50.802783 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":42,"threads_started":1}
I20260807 22:45:50.802870 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:50.799833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.16.98.189:36023)
0807 22:45:50.799944 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:50.799946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:50.799955 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:45:50.799983 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:50.799984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:50.799994 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:50.800072 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:50.800597 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:50.801074 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:50.802062 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:50.802547 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:50.802650 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:50.802660 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:50.802760 (+ 100us) server_negotiation.cc:299] Negotiation successful
0807 22:45:50.802802 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":42,"threads_started":1}
I20260807 22:45:50.802903 23900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b7eb91351014fcd878bb334667f3c43" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0cb3166b76864a979588c5abf24a5ba8" is_pre_election: true
I20260807 22:45:50.803027 23900 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b7eb91351014fcd878bb334667f3c43 in term 1.
I20260807 22:45:50.803193 23685 leader_election.cc:302] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cb3166b76864a979588c5abf24a5ba8, 9b7eb91351014fcd878bb334667f3c43; no voters: 059a0a6fe8314dc4a77f3e93df688221
I20260807 22:45:50.803220 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cb3166b76864a979588c5abf24a5ba8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9b7eb91351014fcd878bb334667f3c43" is_pre_election: true
I20260807 22:45:50.803319 23710 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb3166b76864a979588c5abf24a5ba8 in term 1.
I20260807 22:45:50.803376 24418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 22:45:50.803418 24418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 059a0a6fe8314dc4a77f3e93df688221)
I20260807 22:45:50.803439 24418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:45:50.803458 23874 leader_election.cc:302] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cb3166b76864a979588c5abf24a5ba8, 9b7eb91351014fcd878bb334667f3c43; no voters: 059a0a6fe8314dc4a77f3e93df688221
I20260807 22:45:50.803557 24417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 22:45:50.803617 24417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 059a0a6fe8314dc4a77f3e93df688221)
I20260807 22:45:50.803670 24417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:45:50.804308 24418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } }
I20260807 22:45:50.804453 24418 leader_election.cc:288] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [CANDIDATE]: Term 2 election: Requested vote from peers 059a0a6fe8314dc4a77f3e93df688221 (127.16.98.190:35275), 0cb3166b76864a979588c5abf24a5ba8 (127.16.98.188:35661)
I20260807 22:45:50.804414 24417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } }
I20260807 22:45:50.804554 24417 leader_election.cc:288] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [CANDIDATE]: Term 2 election: Requested vote from peers 059a0a6fe8314dc4a77f3e93df688221 (127.16.98.190:35275), 9b7eb91351014fcd878bb334667f3c43 (127.16.98.189:36023)
I20260807 22:45:50.804658 23900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b7eb91351014fcd878bb334667f3c43" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0cb3166b76864a979588c5abf24a5ba8"
I20260807 22:45:50.804718 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:50.804572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:39484)
0807 22:45:50.804666 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:50.804695 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 22:45:50.804741 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:50.804652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:39500)
0807 22:45:50.804687 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:50.804714 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260807 22:45:50.804765 23900 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9b7eb91351014fcd878bb334667f3c43 in current term 2: Already voted for candidate 0cb3166b76864a979588c5abf24a5ba8 in this term.
I20260807 22:45:50.804816 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cb3166b76864a979588c5abf24a5ba8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9b7eb91351014fcd878bb334667f3c43"
I20260807 22:45:50.804913 23710 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0cb3166b76864a979588c5abf24a5ba8 in current term 2: Already voted for candidate 9b7eb91351014fcd878bb334667f3c43 in this term.
I20260807 22:45:50.805188 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:50.805078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:39510)
0807 22:45:50.805125 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:50.805158 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260807 22:45:50.805258 23684 leader_election.cc:334] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 059a0a6fe8314dc4a77f3e93df688221 (127.16.98.190:35275): Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
I20260807 22:45:50.805263 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:50.805164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:39514)
0807 22:45:50.805218 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:50.805242 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260807 22:45:50.805294 23684 leader_election.cc:302] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b7eb91351014fcd878bb334667f3c43; no voters: 059a0a6fe8314dc4a77f3e93df688221, 0cb3166b76864a979588c5abf24a5ba8
W20260807 22:45:50.805334 23874 leader_election.cc:334] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 059a0a6fe8314dc4a77f3e93df688221 (127.16.98.190:35275): Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
I20260807 22:45:50.805380 23874 leader_election.cc:302] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cb3166b76864a979588c5abf24a5ba8; no voters: 059a0a6fe8314dc4a77f3e93df688221, 9b7eb91351014fcd878bb334667f3c43
I20260807 22:45:50.805377 24418 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 22:45:50.805469 24417 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 22:45:51.295907 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:51.295617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.131:58827)
0807 22:45:51.295816 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:51.295857 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260807 22:45:51.295908 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:51.295617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.129:38505)
0807 22:45:51.295785 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:51.295830 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260807 22:45:51.295943 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:51.295749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.130:43995)
0807 22:45:51.295862 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:51.295897 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260807 22:45:52.296746 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.296537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.131:39811)
0807 22:45:52.296666 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.296698 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260807 22:45:52.296746 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.296402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.129:53327)
0807 22:45:52.296660 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.296698 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":71,"threads_started":1}
I20260807 22:45:52.296741 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.296476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.130:36851)
0807 22:45:52.296619 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.296666 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260807 22:45:52.420368 24433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:52.420493 24433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } }
I20260807 22:45:52.420789 24433 leader_election.cc:288] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 059a0a6fe8314dc4a77f3e93df688221 (127.16.98.190:35275), 9b7eb91351014fcd878bb334667f3c43 (127.16.98.189:36023)
I20260807 22:45:52.421205 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.420990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:32792)
0807 22:45:52.421103 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.421148 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":68,"threads_started":1}
I20260807 22:45:52.421341 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cb3166b76864a979588c5abf24a5ba8" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9b7eb91351014fcd878bb334667f3c43" is_pre_election: true
I20260807 22:45:52.421482 23710 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb3166b76864a979588c5abf24a5ba8 in term 2.
I20260807 22:45:52.421722 23874 leader_election.cc:302] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb3166b76864a979588c5abf24a5ba8, 9b7eb91351014fcd878bb334667f3c43; no voters:
I20260807 22:45:52.421778 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.421655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:32802)
0807 22:45:52.421705 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.421735 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260807 22:45:52.421818 23874 leader_election.cc:334] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 059a0a6fe8314dc4a77f3e93df688221 (127.16.98.190:35275): Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
I20260807 22:45:52.421872 24433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 22:45:52.421933 24433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:52.421970 24433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 2 FOLLOWER]: Advancing to term 3
I20260807 22:45:52.422824 24433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } }
I20260807 22:45:52.422973 24433 leader_election.cc:288] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [CANDIDATE]: Term 3 election: Requested vote from peers 059a0a6fe8314dc4a77f3e93df688221 (127.16.98.190:35275), 9b7eb91351014fcd878bb334667f3c43 (127.16.98.189:36023)
I20260807 22:45:52.423192 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.423099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:32806)
0807 22:45:52.423135 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.423163 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260807 22:45:52.423291 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cb3166b76864a979588c5abf24a5ba8" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9b7eb91351014fcd878bb334667f3c43"
I20260807 22:45:52.423380 23710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 2 FOLLOWER]: Advancing to term 3
I20260807 22:45:52.423538 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.423455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:32814)
0807 22:45:52.423500 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.423524 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260807 22:45:52.423650 23874 leader_election.cc:334] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 059a0a6fe8314dc4a77f3e93df688221 (127.16.98.190:35275): Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
I20260807 22:45:52.424130 23710 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb3166b76864a979588c5abf24a5ba8 in term 3.
I20260807 22:45:52.424304 23874 leader_election.cc:302] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cb3166b76864a979588c5abf24a5ba8, 9b7eb91351014fcd878bb334667f3c43; no voters: 059a0a6fe8314dc4a77f3e93df688221
I20260807 22:45:52.424402 24433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 22:45:52.424536 24433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [term 3 LEADER]: Becoming Leader. State: Replica: 0cb3166b76864a979588c5abf24a5ba8, State: Running, Role: LEADER
I20260807 22:45:52.424633 24433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } }
I20260807 22:45:52.425007 24436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cb3166b76864a979588c5abf24a5ba8. Latest consensus state: current_term: 3 leader_uuid: "0cb3166b76864a979588c5abf24a5ba8" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } } }
I20260807 22:45:52.425098 24436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:52.425282 24438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:52.425696 24438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:52.425794 24438 catalog_manager.cc:1259] Loaded cluster ID: 9d54760e43384d9bbd11c0ae1393fd44
I20260807 22:45:52.425828 24438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:45:52.426023 24438 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:52.426123 24438 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: Loaded TSK: 0
I20260807 22:45:52.426265 24438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:45:52.444964 23889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:45:52.446825 23889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:52.448068 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.447927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:32826)
0807 22:45:52.447994 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.448035 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260807 22:45:52.448232 23710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [term 3 FOLLOWER]: Refusing update from remote peer 0cb3166b76864a979588c5abf24a5ba8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260807 22:45:52.448642 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.448415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.0.0.1:32832)
0807 22:45:52.448468 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.448530 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260807 22:45:52.448518 24436 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260807 22:45:52.448776 23874 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 -> Peer 059a0a6fe8314dc4a77f3e93df688221 (127.16.98.190:35275): Couldn't send request to peer 059a0a6fe8314dc4a77f3e93df688221. Status: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 22:45:52.449435 24440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cb3166b76864a979588c5abf24a5ba8. Latest consensus state: current_term: 3 leader_uuid: "0cb3166b76864a979588c5abf24a5ba8" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } } }
I20260807 22:45:52.449591 24440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:45:52.450048 24433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0cb3166b76864a979588c5abf24a5ba8" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } } }
I20260807 22:45:52.450150 24433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:52.450357 24441 mvcc.cc:205] Tried to move back new op lower bound from 7316040714024509440 to 7316040713931788288. Current Snapshot: MvccSnapshot[applied={T|T < 7316040714024509440}]
I20260807 22:45:52.450552 24440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0cb3166b76864a979588c5abf24a5ba8" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "059a0a6fe8314dc4a77f3e93df688221" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 35275 } } peers { permanent_uuid: "9b7eb91351014fcd878bb334667f3c43" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 36023 } attrs { promote: false } } peers { permanent_uuid: "0cb3166b76864a979588c5abf24a5ba8" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 35661 } attrs { promote: false } } }
I20260807 22:45:52.450646 24440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7eb91351014fcd878bb334667f3c43 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:45:52.455585 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.452050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:42247 (local address 127.0.0.1:53876)
0807 22:45:52.452237 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.452249 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:52.452301 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:52.452922 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:52.452923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:52.452925 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:52.452970 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:52.452972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:52.453807 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:52.453808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:52.454646 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:52.454649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:52.455356 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:52.455357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:52.455422 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:52.455428 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 22:45:52.455458 (+ 30us) client_negotiation.cc:240] Negotiation successful
0807 22:45:52.455498 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":37,"thread_start_us":93,"threads_started":1}
I20260807 22:45:52.455880 24447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.452124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53876 (local address 127.16.98.129:42247)
0807 22:45:52.452530 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:52.452533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:52.452542 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:45:52.452599 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:52.452600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:52.452614 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:52.452687 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:52.453076 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:52.453719 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:52.454715 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:52.455256 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:52.455714 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:52.455731 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:52.455750 (+ 19us) server_negotiation.cc:299] Negotiation successful
0807 22:45:52.455800 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":54,"threads_started":1}
I20260807 22:45:52.455915 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.452005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:43797 (local address 127.0.0.1:34018)
0807 22:45:52.452040 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.452053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:52.452109 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:52.452382 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:52.452383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:52.452385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:52.452440 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:52.452442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:52.453573 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:52.453574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:52.454902 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:52.454904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:52.455644 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:52.455645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:52.455726 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:52.455732 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 22:45:52.455793 (+ 61us) client_negotiation.cc:240] Negotiation successful
0807 22:45:52.455841 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260807 22:45:52.456259 24006 tablet_service.cc:1467] Processing CreateTablet for tablet 276611452c714874bb89577f87fc1ffe (DEFAULT_TABLE table=test-table [id=4880941569074cc7bec9bbd5d793aea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:52.456439 24006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 276611452c714874bb89577f87fc1ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:52.456492 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.452113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34018 (local address 127.16.98.131:43797)
0807 22:45:52.452214 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:52.452216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:52.452228 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:45:52.452258 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:52.452259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:52.452267 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:52.452375 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:52.452523 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:52.453480 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:52.455001 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:52.455556 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:52.455627 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:52.455641 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:52.456395 (+ 754us) server_negotiation.cc:299] Negotiation successful
0807 22:45:52.456438 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":58,"threads_started":1}
I20260807 22:45:52.456743 24280 tablet_service.cc:1467] Processing CreateTablet for tablet 276611452c714874bb89577f87fc1ffe (DEFAULT_TABLE table=test-table [id=4880941569074cc7bec9bbd5d793aea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:52.456873 24280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 276611452c714874bb89577f87fc1ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:52.458261 24450 tablet_bootstrap.cc:492] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a: Bootstrap starting.
I20260807 22:45:52.458408 24451 tablet_bootstrap.cc:492] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968: Bootstrap starting.
I20260807 22:45:52.458892 24451 tablet_bootstrap.cc:654] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:52.458930 24450 tablet_bootstrap.cc:654] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:52.458986 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.452299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:45055 (local address 127.0.0.1:42060)
0807 22:45:52.452432 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.452444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:52.452766 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:52.453044 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:52.453045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:52.453047 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:52.453093 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:52.453095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:52.453931 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:52.453932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:52.455120 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:52.455121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:52.456452 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:52.456453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:52.456517 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:52.456523 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 22:45:52.458811 (+ 2288us) client_negotiation.cc:240] Negotiation successful
0807 22:45:52.458878 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":86,"threads_started":1}
I20260807 22:45:52.459411 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.452508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.16.98.130:45055)
0807 22:45:52.452876 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:52.452878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:52.452887 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:45:52.452913 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:52.452914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:52.452924 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:52.453025 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:52.453169 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:52.453820 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:52.455307 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:52.456298 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:52.459184 (+ 2886us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:52.459205 (+ 21us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:52.459229 (+ 24us) server_negotiation.cc:299] Negotiation successful
0807 22:45:52.459312 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":96,"threads_started":1}
I20260807 22:45:52.459700 24144 tablet_service.cc:1467] Processing CreateTablet for tablet 276611452c714874bb89577f87fc1ffe (DEFAULT_TABLE table=test-table [id=4880941569074cc7bec9bbd5d793aea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:52.459846 24144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 276611452c714874bb89577f87fc1ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:52.459853 24450 tablet_bootstrap.cc:492] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a: No bootstrap required, opened a new log
I20260807 22:45:52.459872 24451 tablet_bootstrap.cc:492] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968: No bootstrap required, opened a new log
I20260807 22:45:52.459897 24450 ts_tablet_manager.cc:1397] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 22:45:52.459934 24451 ts_tablet_manager.cc:1397] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 22:45:52.460067 24451 raft_consensus.cc:348] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } }
I20260807 22:45:52.460063 24450 raft_consensus.cc:348] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } }
I20260807 22:45:52.460297 24450 raft_consensus.cc:374] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:52.460338 24450 raft_consensus.cc:729] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e9c38c98ed4690a9a434fa701ceb3a, State: Initialized, Role: FOLLOWER
I20260807 22:45:52.460330 24451 raft_consensus.cc:374] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:52.460388 24451 raft_consensus.cc:729] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93be69dbc9034b229bbc074bc8fde968, State: Initialized, Role: FOLLOWER
I20260807 22:45:52.460417 24450 consensus_queue.cc:260] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } }
I20260807 22:45:52.460438 24451 consensus_queue.cc:260] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } }
I20260807 22:45:52.460675 24450 ts_tablet_manager.cc:1428] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:52.460680 24451 ts_tablet_manager.cc:1428] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:52.461205 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.461019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.131:59613)
0807 22:45:52.461146 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.461178 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260807 22:45:52.461288 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.461015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.129:39917)
0807 22:45:52.461222 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.461260 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260807 22:45:52.461843 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.461719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.131:50717)
0807 22:45:52.461792 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.461818 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 22:45:52.462023 24072 heartbeater.cc:503] Master 127.16.98.188:35661 was elected leader, sending a full tablet report...
I20260807 22:45:52.462028 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.461900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.129:52583)
0807 22:45:52.461972 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.461998 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 22:45:52.462244 24347 heartbeater.cc:503] Master 127.16.98.188:35661 was elected leader, sending a full tablet report...
I20260807 22:45:52.462862 24456 tablet_bootstrap.cc:492] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a: Bootstrap starting.
I20260807 22:45:52.463285 24456 tablet_bootstrap.cc:654] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:52.463975 24456 tablet_bootstrap.cc:492] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a: No bootstrap required, opened a new log
I20260807 22:45:52.464047 24456 ts_tablet_manager.cc:1397] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:52.464172 24456 raft_consensus.cc:348] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } }
I20260807 22:45:52.464246 24456 raft_consensus.cc:374] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:52.464270 24456 raft_consensus.cc:729] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f89601ecd4ae42c59e1ffbd5bd25ad1a, State: Initialized, Role: FOLLOWER
I20260807 22:45:52.464332 24456 consensus_queue.cc:260] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } }
I20260807 22:45:52.464601 24456 ts_tablet_manager.cc:1428] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:52.465045 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.464914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.130:44285)
0807 22:45:52.464979 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.465017 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260807 22:45:52.465209 24210 heartbeater.cc:503] Master 127.16.98.188:35661 was elected leader, sending a full tablet report...
I20260807 22:45:52.465458 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.465336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.130:52811)
0807 22:45:52.465396 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.465429 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260807 22:45:52.602936 24350 tablet.cc:2369] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:45:52.663187 24458 raft_consensus.cc:482] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:52.663328 24458 raft_consensus.cc:504] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } }
I20260807 22:45:52.663692 24458 leader_election.cc:288] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93be69dbc9034b229bbc074bc8fde968 (127.16.98.131:43797), 57e9c38c98ed4690a9a434fa701ceb3a (127.16.98.129:42247)
I20260807 22:45:52.663933 24026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276611452c714874bb89577f87fc1ffe" candidate_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" is_pre_election: true
I20260807 22:45:52.664004 24301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276611452c714874bb89577f87fc1ffe" candidate_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93be69dbc9034b229bbc074bc8fde968" is_pre_election: true
I20260807 22:45:52.664094 24301 raft_consensus.cc:2454] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f89601ecd4ae42c59e1ffbd5bd25ad1a in term 0.
I20260807 22:45:52.664121 24026 raft_consensus.cc:2454] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f89601ecd4ae42c59e1ffbd5bd25ad1a in term 0.
I20260807 22:45:52.664285 24098 leader_election.cc:302] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93be69dbc9034b229bbc074bc8fde968, f89601ecd4ae42c59e1ffbd5bd25ad1a; no voters:
I20260807 22:45:52.664409 24458 raft_consensus.cc:2781] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:52.664474 24458 raft_consensus.cc:482] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:52.664510 24458 raft_consensus.cc:3037] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:52.665688 24458 raft_consensus.cc:504] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } }
I20260807 22:45:52.666054 24458 leader_election.cc:288] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [CANDIDATE]: Term 1 election: Requested vote from peers 93be69dbc9034b229bbc074bc8fde968 (127.16.98.131:43797), 57e9c38c98ed4690a9a434fa701ceb3a (127.16.98.129:42247)
I20260807 22:45:52.666131 24026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276611452c714874bb89577f87fc1ffe" candidate_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e9c38c98ed4690a9a434fa701ceb3a"
I20260807 22:45:52.666132 24301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276611452c714874bb89577f87fc1ffe" candidate_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93be69dbc9034b229bbc074bc8fde968"
I20260807 22:45:52.666216 24301 raft_consensus.cc:3037] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:52.666216 24026 raft_consensus.cc:3037] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:52.667053 24026 raft_consensus.cc:2454] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f89601ecd4ae42c59e1ffbd5bd25ad1a in term 1.
I20260807 22:45:52.667053 24301 raft_consensus.cc:2454] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f89601ecd4ae42c59e1ffbd5bd25ad1a in term 1.
I20260807 22:45:52.667245 24098 leader_election.cc:302] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93be69dbc9034b229bbc074bc8fde968, f89601ecd4ae42c59e1ffbd5bd25ad1a; no voters:
I20260807 22:45:52.667351 24458 raft_consensus.cc:2781] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:52.667498 24458 raft_consensus.cc:686] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 1 LEADER]: Becoming Leader. State: Replica: f89601ecd4ae42c59e1ffbd5bd25ad1a, State: Running, Role: LEADER
I20260807 22:45:52.667589 24458 consensus_queue.cc:237] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } }
I20260807 22:45:52.668017 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.667879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.130:56599)
0807 22:45:52.667928 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.667981 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260807 22:45:52.668339 23890 catalog_manager.cc:5495] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a reported cstate change: term changed from 0 to 1, leader changed from <none> to f89601ecd4ae42c59e1ffbd5bd25ad1a (127.16.98.130). New cstate: current_term: 1 leader_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f89601ecd4ae42c59e1ffbd5bd25ad1a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45055 } health_report { overall_health: HEALTHY } } }
I20260807 22:45:52.689121 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.685505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.16.98.130:45055)
0807 22:45:52.685576 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:52.685578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:52.685589 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:45:52.685625 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:52.685627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:52.685638 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:52.685724 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:52.685880 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:52.686688 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:52.688042 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:52.688378 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:52.688478 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:52.688785 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:52.688827 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:52.688894 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:52.688998 (+ 104us) server_negotiation.cc:299] Negotiation successful
0807 22:45:52.689028 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 22:45:52.690231 24301 raft_consensus.cc:1261] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968 [term 1 FOLLOWER]: Refusing update from remote peer f89601ecd4ae42c59e1ffbd5bd25ad1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:52.690277 24026 raft_consensus.cc:1261] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a [term 1 FOLLOWER]: Refusing update from remote peer f89601ecd4ae42c59e1ffbd5bd25ad1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:52.690511 24459 consensus_queue.cc:1035] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "93be69dbc9034b229bbc074bc8fde968" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:52.690635 24458 consensus_queue.cc:1035] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "57e9c38c98ed4690a9a434fa701ceb3a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 42247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:52.691402 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.691266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.129:45357)
0807 22:45:52.691321 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.691357 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260807 22:45:52.691859 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.691719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.131:46473)
0807 22:45:52.691793 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.691820 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 22:45:52.693024 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.692834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.130:55063)
0807 22:45:52.692932 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.692985 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 22:45:52.693600 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.693464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.129:59335)
0807 22:45:52.693539 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.693576 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 22:45:52.694515 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.694376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.130:41093)
0807 22:45:52.694456 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.694484 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260807 22:45:52.695318 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.695164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:35275 (local address 127.16.98.131:48203)
0807 22:45:52.695254 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.695290 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:35275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260807 22:45:52.695576 23889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:52.695667 23889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:52.697023 23889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cb3166b76864a979588c5abf24a5ba8: Sending DeleteTablet for 3 replicas of tablet 276611452c714874bb89577f87fc1ffe
I20260807 22:45:52.697278 24006 tablet_service.cc:1514] Processing DeleteTablet for tablet 276611452c714874bb89577f87fc1ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:52 UTC) from {username='oryx', principal='oryx/127.16.98.188@KRBTEST.COM'} at 127.0.0.1:53876
I20260807 22:45:52.697314 24144 tablet_service.cc:1514] Processing DeleteTablet for tablet 276611452c714874bb89577f87fc1ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:52 UTC) from {username='oryx', principal='oryx/127.16.98.188@KRBTEST.COM'} at 127.0.0.1:42060
I20260807 22:45:52.697330 24280 tablet_service.cc:1514] Processing DeleteTablet for tablet 276611452c714874bb89577f87fc1ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:52 UTC) from {username='oryx', principal='oryx/127.16.98.188@KRBTEST.COM'} at 127.0.0.1:34018
I20260807 22:45:52.697549 24470 tablet_replica.cc:330] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a: stopping tablet replica
I20260807 22:45:52.697564 24472 tablet_replica.cc:330] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a: stopping tablet replica
I20260807 22:45:52.697614 24470 raft_consensus.cc:2229] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:52.697619 24472 raft_consensus.cc:2229] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:45:52.697677 24472 raft_consensus.cc:2258] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:52.697672 24470 raft_consensus.cc:2258] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:52.698027 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 23850
I20260807 22:45:52.698027 24472 ts_tablet_manager.cc:1905] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:52.698150 24470 ts_tablet_manager.cc:1905] T 276611452c714874bb89577f87fc1ffe P 57e9c38c98ed4690a9a434fa701ceb3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:52.700644 24472 ts_tablet_manager.cc:1918] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:52.700709 24472 log.cc:1196] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/276611452c714874bb89577f87fc1ffe
I20260807 22:45:52.700976 24472 ts_tablet_manager.cc:1939] T 276611452c714874bb89577f87fc1ffe P f89601ecd4ae42c59e1ffbd5bd25ad1a: Deleting consensus metadata
I20260807 22:45:52.701541 23874 catalog_manager.cc:4841] TS f89601ecd4ae42c59e1ffbd5bd25ad1a (127.16.98.130:45055): tablet 276611452c714874bb89577f87fc1ffe (table test-table [id=4880941569074cc7bec9bbd5d793aea1]) successfully deleted
I20260807 22:45:52.707031 24471 tablet_replica.cc:330] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968: stopping tablet replica
I20260807 22:45:52.707136 24471 raft_consensus.cc:2229] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:52.707177 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:52.706914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:42247 (local address 127.0.0.1:53878)
0807 22:45:52.707086 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:52.707130 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260807 22:45:52.707165 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 24080
I20260807 22:45:52.707206 24471 raft_consensus.cc:2258] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 22:45:52.707356 23874 catalog_manager.cc:4584] TS 57e9c38c98ed4690a9a434fa701ceb3a (127.16.98.129:42247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 276611452c714874bb89577f87fc1ffe: Network error: Client connection negotiation failed: client connection to 127.16.98.129:42247: connect: Connection refused (error 111)
I20260807 22:45:52.708300 24471 ts_tablet_manager.cc:1905] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:52.710795 24471 ts_tablet_manager.cc:1918] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:52.710896 24471 log.cc:1196] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/276611452c714874bb89577f87fc1ffe
I20260807 22:45:52.711155 24471 ts_tablet_manager.cc:1939] T 276611452c714874bb89577f87fc1ffe P 93be69dbc9034b229bbc074bc8fde968: Deleting consensus metadata
I20260807 22:45:52.711757 23875 catalog_manager.cc:4841] TS 93be69dbc9034b229bbc074bc8fde968 (127.16.98.131:43797): tablet 276611452c714874bb89577f87fc1ffe (table test-table [id=4880941569074cc7bec9bbd5d793aea1]) successfully deleted
I20260807 22:45:52.717509 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 24217
I20260807 22:45:52.725088 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 23562
I20260807 22:45:52.731532 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 23754
2026-08-07T22:45:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6844 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:45:52 dist-test-slave-rgb1 krb5kdc[24475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:45:52 dist-test-slave-rgb1 krb5kdc[24475](info): set up 2 sockets
Aug 07 22:45:52 dist-test-slave-rgb1 krb5kdc[24475](info): commencing operation
krb5kdc: starting...
W20260807 22:45:53.870954 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:45:53 dist-test-slave-rgb1 krb5kdc[24475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:45:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.190.keytab.
Entry for principal oryx/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.190.keytab.
I20260807 22:45:54.030133 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:36419
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.190.keytab
--principal=oryx/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:36419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:54.119450 24488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:54.119766 24488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:54.119827 24488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:54.121210 24488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:54.121296 24488 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 22:45:54.121322 24488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:54.121346 24488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:54.121407 24488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:54.121443 24488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:54.123236 24488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:36419
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.190.keytab
--principal=oryx/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:54.123565 24488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:54.123845 24488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:54.129554 24494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:54.129524 24493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:54.129554 24496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:54.129923 24488 server_base.cc:1034] running on GCE node
I20260807 22:45:54.130256 24488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:54.130616 24488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:54.131805 24488 hybrid_clock.cc:648] HybridClock initialized: now 1786142754131775 us; error 45 us; skew 500 ppm
Aug 07 22:45:54 dist-test-slave-rgb1 krb5kdc[24475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142754, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:54.134984 24488 init.cc:377] Logged in from keytab as oryx/127.16.98.190@KRBTEST.COM (short username oryx)
I20260807 22:45:54.135624 24488 webserver.cc:460] Webserver started at http://127.16.98.190:40731/ using document root <none> and password file <none>
I20260807 22:45:54.135916 24488 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:54.135970 24488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:54.136171 24488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:54.137578 24488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "08c5e1345a704743aa11e1f790451afd"
format_stamp: "Formatted at 2026-08-07 22:45:54 on dist-test-slave-rgb1"
server_key: "d2b7b5ecd34d04b227b4c53ed92f3af4"
server_key_iv: "3ecb97f0778a4123d63690f37735963c"
server_key_version: "encryptionkey@0"
I20260807 22:45:54.137988 24488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "08c5e1345a704743aa11e1f790451afd"
format_stamp: "Formatted at 2026-08-07 22:45:54 on dist-test-slave-rgb1"
server_key: "d2b7b5ecd34d04b227b4c53ed92f3af4"
server_key_iv: "3ecb97f0778a4123d63690f37735963c"
server_key_version: "encryptionkey@0"
I20260807 22:45:54.139628 24488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:45:54.140707 24503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:54.141059 24488 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:54.141218 24488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "08c5e1345a704743aa11e1f790451afd"
format_stamp: "Formatted at 2026-08-07 22:45:54 on dist-test-slave-rgb1"
server_key: "d2b7b5ecd34d04b227b4c53ed92f3af4"
server_key_iv: "3ecb97f0778a4123d63690f37735963c"
server_key_version: "encryptionkey@0"
I20260807 22:45:54.141317 24488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:54.154780 24488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:54.157719 24488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:54.157999 24488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:54.164127 24488 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:36419
I20260807 22:45:54.164189 24566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:36419 every 8 connection(s)
I20260807 22:45:54.164614 24488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:54.165364 24567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:54.166879 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 24488
I20260807 22:45:54.166982 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:54.167236 16778 external_mini_cluster.cc:1428] Setting key f89d9fc6f9672e980d9eef14f30510de
I20260807 22:45:54.168301 24567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd: Bootstrap starting.
I20260807 22:45:54.169070 24567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:54.169443 24567 log.cc:824] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:54.170552 24567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd: No bootstrap required, opened a new log
I20260807 22:45:54.173290 24567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c5e1345a704743aa11e1f790451afd" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36419 } }
I20260807 22:45:54.173488 24567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:54.173538 24567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c5e1345a704743aa11e1f790451afd, State: Initialized, Role: FOLLOWER
I20260807 22:45:54.173705 24567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c5e1345a704743aa11e1f790451afd" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36419 } }
I20260807 22:45:54.173792 24567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:54.173834 24567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:54.173889 24567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:54.174523 24567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c5e1345a704743aa11e1f790451afd" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36419 } }
I20260807 22:45:54.174651 24567 leader_election.cc:302] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08c5e1345a704743aa11e1f790451afd; no voters:
I20260807 22:45:54.174872 24567 leader_election.cc:288] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:54.174991 24572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:54.175186 24567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:54.175199 24572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [term 1 LEADER]: Becoming Leader. State: Replica: 08c5e1345a704743aa11e1f790451afd, State: Running, Role: LEADER
Aug 07 22:45:54 dist-test-slave-rgb1 krb5kdc[24475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
I20260807 22:45:54.175480 24572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c5e1345a704743aa11e1f790451afd" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36419 } }
I20260807 22:45:54.176052 24573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08c5e1345a704743aa11e1f790451afd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c5e1345a704743aa11e1f790451afd" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36419 } } }
I20260807 22:45:54.176164 24573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:54.177524 24574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08c5e1345a704743aa11e1f790451afd. Latest consensus state: current_term: 1 leader_uuid: "08c5e1345a704743aa11e1f790451afd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08c5e1345a704743aa11e1f790451afd" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36419 } } }
W20260807 22:45:54.177569 24588 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260807 22:45:54.177654 24574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd [sys.catalog]: This master's current role is: LEADER
W20260807 22:45:54.177719 24588 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:45:54.177846 24581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:54.178041 24581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:54.179564 24581 catalog_manager.cc:1347] Generated new cluster ID: 306b8d994ca4497d9c2aaf076812d079
I20260807 22:45:54.179618 24581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:45:54.181201 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.167903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.16.98.190:36419)
0807 22:45:54.168140 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:54.168150 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:54.168187 (+ 37us) server_negotiation.cc:407] Connection header received
0807 22:45:54.168721 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:54.168728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:54.168777 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:54.168926 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:54.169677 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.171147 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.172163 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.172838 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.172979 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.176236 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:54.176240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:54.176241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:54.176254 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:54.179024 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.179535 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.179539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.179539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.179658 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.179918 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.179919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.179919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.180295 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:54.180465 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:54.180646 (+ 181us) server_negotiation.cc:299] Negotiation successful
0807 22:45:54.180947 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":114,"threads_started":1}
I20260807 22:45:54.192147 24581 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:54.192977 24581 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:54.197110 24581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd: Generated new TSK 0
I20260807 22:45:54.197312 24581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.129.keytab.
Entry for principal oryx/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.129.keytab.
I20260807 22:45:54.236830 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36419
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.129.keytab
--principal=oryx/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:54.327171 24593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:54.327410 24593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:54.327441 24593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:54.328848 24593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:54.328897 24593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:54.328923 24593 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:45:54.328944 24593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:45:54.330775 24593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.129.keytab
--principal=oryx/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.98.190:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:54.331070 24593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:54.331342 24593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:54.337311 24599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:54.337311 24598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:54.337350 24601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:54.337604 24593 server_base.cc:1034] running on GCE node
I20260807 22:45:54.337873 24593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:54.338104 24593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:54.339498 24593 hybrid_clock.cc:648] HybridClock initialized: now 1786142754339483 us; error 30 us; skew 500 ppm
Aug 07 22:45:54 dist-test-slave-rgb1 krb5kdc[24475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142754, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:54.342551 24593 init.cc:377] Logged in from keytab as oryx/127.16.98.129@KRBTEST.COM (short username oryx)
I20260807 22:45:54.343237 24593 webserver.cc:460] Webserver started at http://127.16.98.129:39123/ using document root <none> and password file <none>
I20260807 22:45:54.343509 24593 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:54.343561 24593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:54.343716 24593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:54.345088 24593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "a6034f1118ee42ccb46aee7eeb4cdde1"
format_stamp: "Formatted at 2026-08-07 22:45:54 on dist-test-slave-rgb1"
server_key: "939b423495d6b35fedf1b0781963955e"
server_key_iv: "2c8572570b600eceba656c5f417182b4"
server_key_version: "encryptionkey@0"
I20260807 22:45:54.345496 24593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "a6034f1118ee42ccb46aee7eeb4cdde1"
format_stamp: "Formatted at 2026-08-07 22:45:54 on dist-test-slave-rgb1"
server_key: "939b423495d6b35fedf1b0781963955e"
server_key_iv: "2c8572570b600eceba656c5f417182b4"
server_key_version: "encryptionkey@0"
I20260807 22:45:54.347110 24593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:45:54.348227 24608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:54.348505 24593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:54.348676 24593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "a6034f1118ee42ccb46aee7eeb4cdde1"
format_stamp: "Formatted at 2026-08-07 22:45:54 on dist-test-slave-rgb1"
server_key: "939b423495d6b35fedf1b0781963955e"
server_key_iv: "2c8572570b600eceba656c5f417182b4"
server_key_version: "encryptionkey@0"
I20260807 22:45:54.348773 24593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:54.388628 24593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:54.391342 24593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:54.391562 24593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:54.392227 24593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:54.392306 24593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:54.392356 24593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:54.392392 24593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 22:45:54 dist-test-slave-rgb1 krb5kdc[24475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142754, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
I20260807 22:45:54.401729 24593 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:41199
I20260807 22:45:54.401835 24723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:41199 every 8 connection(s)
I20260807 22:45:54.402355 24593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:45:54.403937 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 24593
I20260807 22:45:54.404042 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.393065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36419 (local address 127.16.98.129:38127)
0807 22:45:54.393258 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:54.393300 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:54.393900 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:54.394201 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:54.394205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:54.394547 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:54.394762 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.394768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.395704 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.395705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.396742 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.396746 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.397184 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.397185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.397339 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.397976 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:54.397985 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:54.400137 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:54.402312 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:54.402321 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:54.402323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:54.402658 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:54.402936 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:54.402938 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:54.402938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:54.403101 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:54.403423 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:54.403428 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:54.403564 (+ 136us) client_negotiation.cc:769] Sending connection context
0807 22:45:54.403680 (+ 116us) client_negotiation.cc:240] Negotiation successful
0807 22:45:54.403828 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260807 22:45:54.404067 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:45:54.404181 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.393116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:38127 (local address 127.16.98.190:36419)
0807 22:45:54.393190 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:54.393192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:54.393402 (+ 210us) server_negotiation.cc:407] Connection header received
0807 22:45:54.394036 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:54.394038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:54.394053 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:54.394112 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:54.394873 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.395591 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.396844 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.397080 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.397169 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.400268 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:54.400270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:54.400270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:54.400288 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:54.402208 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.402788 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.402790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.402790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.402863 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.403196 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.403196 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.403197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.403323 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:54.403386 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:54.403835 (+ 449us) server_negotiation.cc:299] Negotiation successful
0807 22:45:54.403973 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 22:45:54.404350 16778 external_mini_cluster.cc:1428] Setting key b9b1681ebffc9975c7db9a523349bf74
I20260807 22:45:54.404995 24724 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36419
I20260807 22:45:54.405117 24724 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:54.405339 24724 heartbeater.cc:511] Master 127.16.98.190:36419 requested a full tablet report, sending...
I20260807 22:45:54.405922 24518 ts_manager.cc:194] Registered new tserver with Master: a6034f1118ee42ccb46aee7eeb4cdde1 (127.16.98.129:41199)
I20260807 22:45:54.406636 24518 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:38127
I20260807 22:45:54.414124 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.407087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:34529 (local address 127.16.98.190:36419)
0807 22:45:54.407136 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:54.407138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:54.407226 (+ 88us) server_negotiation.cc:407] Connection header received
0807 22:45:54.407333 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:54.407334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:54.407346 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:54.407385 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:54.407947 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.408459 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.409378 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.409603 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.409690 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.410998 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:54.410999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:54.411000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:54.411007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:54.412781 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.413204 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.413206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.413206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.413275 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.413520 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.413521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.413522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.413645 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:54.413728 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:54.413928 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 22:45:54.414016 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 22:45:54.414139 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.407006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36419 (local address 127.16.98.129:34529)
0807 22:45:54.407163 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:54.407176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:54.407242 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:54.407466 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:54.407467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:54.407765 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:54.407856 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.407859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.408629 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.408630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.409261 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.409264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.409709 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.409711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.409804 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.410371 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:54.410377 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:54.410881 (+ 504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:54.412894 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:54.412895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:54.412896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:54.413098 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:54.413370 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:54.413371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:54.413372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:54.413446 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:54.413734 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:54.413735 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:54.413787 (+ 52us) client_negotiation.cc:769] Sending connection context
0807 22:45:54.413894 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 22:45:54.414020 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for oryx/127.16.98.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.130.keytab.
Entry for principal oryx/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.130.keytab.
I20260807 22:45:54.440479 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36419
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.130.keytab
--principal=oryx/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:54.529955 24733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:54.530228 24733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:54.530272 24733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:54.531494 24733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:54.531569 24733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:54.531605 24733 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:45:54.531684 24733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:45:54.533447 24733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.130.keytab
--principal=oryx/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.98.190:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:54.533730 24733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:54.533998 24733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:54.539415 24738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:54.539433 24739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:54.539422 24741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:54.539673 24733 server_base.cc:1034] running on GCE node
I20260807 22:45:54.539889 24733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:54.540091 24733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:54.541263 24733 hybrid_clock.cc:648] HybridClock initialized: now 1786142754541241 us; error 37 us; skew 500 ppm
Aug 07 22:45:54 dist-test-slave-rgb1 krb5kdc[24475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142754, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:54.544148 24733 init.cc:377] Logged in from keytab as oryx/127.16.98.130@KRBTEST.COM (short username oryx)
I20260807 22:45:54.544871 24733 webserver.cc:460] Webserver started at http://127.16.98.130:38999/ using document root <none> and password file <none>
I20260807 22:45:54.545145 24733 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:54.545197 24733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:54.545349 24733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:54.546645 24733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "68aa9dace3cd4d479522e301c9245599"
format_stamp: "Formatted at 2026-08-07 22:45:54 on dist-test-slave-rgb1"
server_key: "49ea2f28b64a3a00a69c00be691d6bb0"
server_key_iv: "473cf1ee0adc95a7d17fb72da5aafffc"
server_key_version: "encryptionkey@0"
I20260807 22:45:54.547041 24733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "68aa9dace3cd4d479522e301c9245599"
format_stamp: "Formatted at 2026-08-07 22:45:54 on dist-test-slave-rgb1"
server_key: "49ea2f28b64a3a00a69c00be691d6bb0"
server_key_iv: "473cf1ee0adc95a7d17fb72da5aafffc"
server_key_version: "encryptionkey@0"
I20260807 22:45:54.548820 24733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 22:45:54.549813 24748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:54.550104 24733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:54.550220 24733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "68aa9dace3cd4d479522e301c9245599"
format_stamp: "Formatted at 2026-08-07 22:45:54 on dist-test-slave-rgb1"
server_key: "49ea2f28b64a3a00a69c00be691d6bb0"
server_key_iv: "473cf1ee0adc95a7d17fb72da5aafffc"
server_key_version: "encryptionkey@0"
I20260807 22:45:54.550304 24733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:54.565333 24733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:54.568079 24733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:54.568300 24733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:54.569085 24733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:54.569175 24733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:54.569227 24733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:54.569258 24733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:54.577764 24733 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:34781
I20260807 22:45:54.577858 24867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:34781 every 8 connection(s)
I20260807 22:45:54.578197 24733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
Aug 07 22:45:54 dist-test-slave-rgb1 krb5kdc[24475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142754, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
I20260807 22:45:54.581997 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.569486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:36749 (local address 127.16.98.190:36419)
0807 22:45:54.569553 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:54.569556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:54.569859 (+ 303us) server_negotiation.cc:407] Connection header received
0807 22:45:54.570434 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:54.570436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:54.570452 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:54.570554 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:54.571211 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.572037 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.574346 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.574683 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.574770 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.579061 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:54.579062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:54.579063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:54.579071 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:54.580472 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.580967 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.580968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.580969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.581043 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.581316 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.581316 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.581317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.581443 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:54.581523 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:54.581808 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 22:45:54.581895 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260807 22:45:54.582211 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.569533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36419 (local address 127.16.98.130:36749)
0807 22:45:54.569710 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:54.569753 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:54.570318 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:54.570604 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:54.570608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:54.570891 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:54.571110 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.571114 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.572154 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.572156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.574185 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.574189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.574851 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.574854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.575133 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.576975 (+ 1842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:54.576989 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:54.578945 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:54.580607 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:54.580614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:54.580616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:54.580846 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:54.581126 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:54.581127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:54.581127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:54.581238 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:54.581515 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:54.581523 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:54.581667 (+ 144us) client_negotiation.cc:769] Sending connection context
0807 22:45:54.581775 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 22:45:54.581950 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260807 22:45:54.582793 24868 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36419
I20260807 22:45:54.582930 24868 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:54.583135 24868 heartbeater.cc:511] Master 127.16.98.190:36419 requested a full tablet report, sending...
I20260807 22:45:54.583622 24519 ts_manager.cc:194] Registered new tserver with Master: 68aa9dace3cd4d479522e301c9245599 (127.16.98.130:34781)
I20260807 22:45:54.584136 24519 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:36749
I20260807 22:45:54.588469 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 24733
I20260807 22:45:54.588649 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:45:54.588901 16778 external_mini_cluster.cc:1428] Setting key 63c005029c60102a8cb62a944337419a
I20260807 22:45:54.590911 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.583895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:40895 (local address 127.16.98.190:36419)
0807 22:45:54.583930 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:54.583932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:54.584410 (+ 478us) server_negotiation.cc:407] Connection header received
0807 22:45:54.584460 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:54.584461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:54.584472 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:54.584592 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:54.585063 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.585585 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.586323 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.586530 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.586595 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.587834 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:54.587836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:54.587836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:54.587844 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:54.589689 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.590042 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.590043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.590044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.590101 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.590321 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.590322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.590322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.590475 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:54.590575 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:54.590723 (+ 148us) server_negotiation.cc:299] Negotiation successful
0807 22:45:54.590808 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260807 22:45:54.590922 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.583839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36419 (local address 127.16.98.130:40895)
0807 22:45:54.584328 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:54.584342 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:54.584413 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:54.584613 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:54.584614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:54.584832 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:54.584907 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.584909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.585690 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.585691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.586221 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.586223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.586614 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.586615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.586672 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.587240 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:54.587246 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:54.587733 (+ 487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:54.589777 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:54.589778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:54.589779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:54.589964 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:54.590178 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:54.590179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:54.590179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:54.590241 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:54.590556 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:54.590557 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:54.590607 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 22:45:54.590694 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 22:45:54.590808 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.16.98.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.131.keytab.
Entry for principal oryx/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.131.keytab.
I20260807 22:45:54.620630 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36419
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.131.keytab
--principal=oryx/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:54.709698 24878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:54.709924 24878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:54.709950 24878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:54.711186 24878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:54.711234 24878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:54.711256 24878 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:45:54.711275 24878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:45:54.713044 24878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.131.keytab
--principal=oryx/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.98.190:36419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:54.713323 24878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:54.713574 24878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:54.718977 24884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:54.718999 24883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:54.719044 24886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:54.719079 24878 server_base.cc:1034] running on GCE node
I20260807 22:45:54.719528 24878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:54.719841 24878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:54.721058 24878 hybrid_clock.cc:648] HybridClock initialized: now 1786142754721028 us; error 47 us; skew 500 ppm
Aug 07 22:45:54 dist-test-slave-rgb1 krb5kdc[24475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142754, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:54.724020 24878 init.cc:377] Logged in from keytab as oryx/127.16.98.131@KRBTEST.COM (short username oryx)
I20260807 22:45:54.724524 24878 webserver.cc:460] Webserver started at http://127.16.98.131:36387/ using document root <none> and password file <none>
I20260807 22:45:54.724797 24878 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:54.724869 24878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:54.725026 24878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:54.726140 24878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "a8054e282ef845688c49c3d68b0fb708"
format_stamp: "Formatted at 2026-08-07 22:45:54 on dist-test-slave-rgb1"
server_key: "d8bab7df318b90e925a64158f3207763"
server_key_iv: "c1de03c24cb44f43c879675227014e0a"
server_key_version: "encryptionkey@0"
I20260807 22:45:54.726513 24878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "a8054e282ef845688c49c3d68b0fb708"
format_stamp: "Formatted at 2026-08-07 22:45:54 on dist-test-slave-rgb1"
server_key: "d8bab7df318b90e925a64158f3207763"
server_key_iv: "c1de03c24cb44f43c879675227014e0a"
server_key_version: "encryptionkey@0"
I20260807 22:45:54.727998 24878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:45:54.728860 24894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:54.729056 24878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:54.729167 24878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "a8054e282ef845688c49c3d68b0fb708"
format_stamp: "Formatted at 2026-08-07 22:45:54 on dist-test-slave-rgb1"
server_key: "d8bab7df318b90e925a64158f3207763"
server_key_iv: "c1de03c24cb44f43c879675227014e0a"
server_key_version: "encryptionkey@0"
I20260807 22:45:54.729243 24878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:54.747222 24878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:54.749639 24878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:54.749809 24878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:54.750360 24878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:54.750422 24878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:54.750499 24878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:54.750532 24878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:54.758056 24878 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:44647
I20260807 22:45:54.758148 25012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:44647 every 8 connection(s)
Aug 07 22:45:54 dist-test-slave-rgb1 krb5kdc[24475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142754, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
I20260807 22:45:54.758500 24878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:45:54.762048 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.751074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:50003 (local address 127.16.98.190:36419)
0807 22:45:54.751136 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:54.751138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:54.751460 (+ 322us) server_negotiation.cc:407] Connection header received
0807 22:45:54.752179 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:54.752181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:54.752196 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:54.752262 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:54.753445 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.753992 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.755565 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.755767 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.755854 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.758925 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:54.758927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:54.758927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:54.758960 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:54.760292 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.760802 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.760803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.760804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.760936 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.761267 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.761268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.761268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.761409 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:54.761475 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:54.761829 (+ 354us) server_negotiation.cc:299] Negotiation successful
0807 22:45:54.761913 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 22:45:54.762161 24903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.751166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36419 (local address 127.16.98.131:50003)
0807 22:45:54.751333 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:54.751369 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:54.752029 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:54.752372 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:54.752397 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:54.752891 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:54.753322 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.753330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.754137 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.754138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.755421 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.755424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.755891 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.755893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.756032 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.756760 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:54.756770 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:54.758789 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:54.760376 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:54.760383 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:54.760385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:54.760699 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:54.761050 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:54.761051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:54.761051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:54.761155 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:54.761510 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:54.761518 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:54.761670 (+ 152us) client_negotiation.cc:769] Sending connection context
0807 22:45:54.761771 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 22:45:54.761934 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260807 22:45:54.762720 25013 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36419
I20260807 22:45:54.762840 25013 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:54.763043 25013 heartbeater.cc:511] Master 127.16.98.190:36419 requested a full tablet report, sending...
I20260807 22:45:54.763477 24518 ts_manager.cc:194] Registered new tserver with Master: a8054e282ef845688c49c3d68b0fb708 (127.16.98.131:44647)
I20260807 22:45:54.764122 24518 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:50003
I20260807 22:45:54.767405 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 24878
I20260807 22:45:54.767504 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:45:54.767751 16778 external_mini_cluster.cc:1428] Setting key f2909df51ba1bac30f8c6b72d90a5d49
I20260807 22:45:54.768659 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:45:54.770524 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.764017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:42701 (local address 127.16.98.190:36419)
0807 22:45:54.764056 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:54.764058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:54.764200 (+ 142us) server_negotiation.cc:407] Connection header received
0807 22:45:54.764302 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:54.764303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:54.764316 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:54.764397 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:54.764809 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.765439 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.766272 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.766483 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.766550 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.767558 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:54.767559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:54.767560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:54.767567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:54.769206 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.769612 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.769613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.769614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.769668 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.769980 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.769981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.769981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.770107 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:54.770177 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:54.770377 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 22:45:54.770451 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260807 22:45:54.770740 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.763955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36419 (local address 127.16.98.131:42701)
0807 22:45:54.764128 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:54.764138 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:54.764216 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:54.764399 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:54.764400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:54.764663 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:54.764726 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.764728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.765540 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.765541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.766113 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.766116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.766557 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.766558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.766607 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.767117 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:54.767122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:54.767406 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:54.769291 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:54.769292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:54.769293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:54.769534 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:54.769735 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:54.769735 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:54.769736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:54.769810 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:54.770185 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:54.770186 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:54.770239 (+ 53us) client_negotiation.cc:769] Sending connection context
0807 22:45:54.770315 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 22:45:54.770445 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
Aug 07 22:45:54 dist-test-slave-rgb1 krb5kdc[24475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142754, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 22:45:54 dist-test-slave-rgb1 krb5kdc[24475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:45:54 dist-test-slave-rgb1 krb5kdc[24475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
W20260807 22:45:54.794047 24570 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:54.787146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.16.98.190:36419)
0807 22:45:54.787462 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:54.787468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:54.787479 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:45:54.787508 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:54.787508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:54.787519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:54.787581 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:54.788402 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.788997 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.789875 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.790083 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.790153 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.793955 (+ 3802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54126: BlockingRecv error: recv got EOF from 127.0.0.1:54126 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 07 22:45:54 dist-test-slave-rgb1 krb5kdc[24475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142754, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
I20260807 22:45:54.803532 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.795099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.16.98.190:36419)
0807 22:45:54.795159 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:54.795162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:54.795255 (+ 93us) server_negotiation.cc:407] Connection header received
0807 22:45:54.795310 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:54.795311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:54.795320 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:54.795368 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:54.796071 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.796550 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.797680 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.798119 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.798195 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.800679 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:54.800680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:54.800681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:54.800688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:54.802145 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.802517 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.802518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.802519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.802582 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:54.802845 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:54.802847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:54.802847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:54.802974 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:54.803066 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:54.803312 (+ 246us) server_negotiation.cc:299] Negotiation successful
0807 22:45:54.803390 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260807 22:45:54.806100 24518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:45:54.807869 24518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:54.816497 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.811569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:44647 (local address 127.0.0.1:54794)
0807 22:45:54.812107 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:54.812120 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:54.812438 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:54.812688 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:54.812689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:54.812691 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:54.812758 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.812760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.813781 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.813781 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.815245 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.815248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.816054 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.816056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.816150 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.816167 (+ 17us) client_negotiation.cc:769] Sending connection context
0807 22:45:54.816290 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 22:45:54.816355 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":33,"thread_start_us":100,"threads_started":1}
I20260807 22:45:54.816639 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.811569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:34781 (local address 127.0.0.1:59656)
0807 22:45:54.812025 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:54.812071 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:54.812189 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:54.812506 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:54.812509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:54.812512 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:54.812631 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.812634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.813612 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.813613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.815246 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.815248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.816300 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.816301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.816361 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.816364 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:45:54.816435 (+ 71us) client_negotiation.cc:240] Negotiation successful
0807 22:45:54.816477 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":55,"thread_start_us":93,"threads_started":1}
I20260807 22:45:54.816957 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.811747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.16.98.131:44647)
0807 22:45:54.811957 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:54.811962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:54.812206 (+ 244us) server_negotiation.cc:407] Connection header received
0807 22:45:54.812526 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:54.812530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:54.812552 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:54.812692 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:54.812861 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.813651 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.815377 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.815940 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.816052 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.816145 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:54.816779 (+ 634us) server_negotiation.cc:299] Negotiation successful
0807 22:45:54.816832 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":94,"threads_started":1}
I20260807 22:45:54.817262 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.811747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.16.98.130:34781)
0807 22:45:54.812035 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:54.812038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:54.812283 (+ 245us) server_negotiation.cc:407] Connection header received
0807 22:45:54.812332 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:54.812334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:54.812353 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:54.812426 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:54.812722 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.813512 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.815345 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.816179 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.816264 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.816341 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:54.816455 (+ 114us) server_negotiation.cc:299] Negotiation successful
0807 22:45:54.816500 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":89,"threads_started":1}
I20260807 22:45:54.817890 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.811849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:41199 (local address 127.0.0.1:48140)
0807 22:45:54.812268 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:54.812278 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:54.812361 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:54.813612 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:54.813613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:54.813616 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:54.813651 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.813652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.815124 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.815126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.816365 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.816369 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.817619 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.817620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.817688 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.817693 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:45:54.817759 (+ 66us) client_negotiation.cc:240] Negotiation successful
0807 22:45:54.817805 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":183,"threads_started":1}
I20260807 22:45:54.817718 24946 tablet_service.cc:1467] Processing CreateTablet for tablet 172deaf9b84b46db987b2df619278a4c (DEFAULT_TABLE table=test-table [id=96550447549e45aeaaaed0f783007db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:54.817965 24801 tablet_service.cc:1467] Processing CreateTablet for tablet 172deaf9b84b46db987b2df619278a4c (DEFAULT_TABLE table=test-table [id=96550447549e45aeaaaed0f783007db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:54.818169 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.811990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48140 (local address 127.16.98.129:41199)
0807 22:45:54.812201 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:54.812205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:54.812352 (+ 147us) server_negotiation.cc:407] Connection header received
0807 22:45:54.812721 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:54.812724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:54.812743 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:54.812833 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:54.813734 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.815029 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.816481 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.817404 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.817497 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.817579 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:54.817795 (+ 216us) server_negotiation.cc:299] Negotiation successful
0807 22:45:54.817842 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":118,"threads_started":1}
I20260807 22:45:54.818187 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 172deaf9b84b46db987b2df619278a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:54.818372 24801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 172deaf9b84b46db987b2df619278a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:54.821734 25042 tablet_bootstrap.cc:492] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708: Bootstrap starting.
I20260807 22:45:54.822052 25043 tablet_bootstrap.cc:492] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599: Bootstrap starting.
I20260807 22:45:54.822497 24658 tablet_service.cc:1467] Processing CreateTablet for tablet 172deaf9b84b46db987b2df619278a4c (DEFAULT_TABLE table=test-table [id=96550447549e45aeaaaed0f783007db5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:54.822669 25042 tablet_bootstrap.cc:654] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:54.822943 24658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 172deaf9b84b46db987b2df619278a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:54.823032 25042 log.cc:824] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:54.823171 25043 tablet_bootstrap.cc:654] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:54.823556 25043 log.cc:824] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:54.823956 25042 tablet_bootstrap.cc:492] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708: No bootstrap required, opened a new log
I20260807 22:45:54.824036 25042 ts_tablet_manager.cc:1397] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:45:54.824229 25043 tablet_bootstrap.cc:492] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599: No bootstrap required, opened a new log
I20260807 22:45:54.824272 25043 ts_tablet_manager.cc:1397] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:45:54.825747 25043 raft_consensus.cc:348] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:54.825917 25043 raft_consensus.cc:374] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:54.825965 25043 raft_consensus.cc:729] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aa9dace3cd4d479522e301c9245599, State: Initialized, Role: FOLLOWER
I20260807 22:45:54.826088 25043 consensus_queue.cc:260] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:54.826377 25043 ts_tablet_manager.cc:1428] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:45:54.826411 24868 heartbeater.cc:503] Master 127.16.98.190:36419 was elected leader, sending a full tablet report...
I20260807 22:45:54.826485 25042 raft_consensus.cc:348] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:54.826638 25042 raft_consensus.cc:374] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:54.826684 25042 raft_consensus.cc:729] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8054e282ef845688c49c3d68b0fb708, State: Initialized, Role: FOLLOWER
I20260807 22:45:54.826807 25042 consensus_queue.cc:260] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:54.827122 25042 ts_tablet_manager.cc:1428] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:45:54.827160 25013 heartbeater.cc:503] Master 127.16.98.190:36419 was elected leader, sending a full tablet report...
I20260807 22:45:54.827617 25049 tablet_bootstrap.cc:492] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1: Bootstrap starting.
I20260807 22:45:54.828363 25049 tablet_bootstrap.cc:654] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1: Neither blocks nor log segments found. Creating new log.
W20260807 22:45:54.828719 24869 tablet.cc:2369] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:45:54.828785 25049 log.cc:824] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:54.829607 25049 tablet_bootstrap.cc:492] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1: No bootstrap required, opened a new log
I20260807 22:45:54.829679 25049 ts_tablet_manager.cc:1397] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:45:54.831301 25049 raft_consensus.cc:348] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:54.831444 25049 raft_consensus.cc:374] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:54.831468 25049 raft_consensus.cc:729] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6034f1118ee42ccb46aee7eeb4cdde1, State: Initialized, Role: FOLLOWER
I20260807 22:45:54.831605 25049 consensus_queue.cc:260] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:54.831988 25049 ts_tablet_manager.cc:1428] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 22:45:54.832149 24724 heartbeater.cc:503] Master 127.16.98.190:36419 was elected leader, sending a full tablet report...
W20260807 22:45:54.903585 24725 tablet.cc:2369] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:45:54.908360 25051 raft_consensus.cc:482] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:54.908535 25051 raft_consensus.cc:504] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:54.908998 25051 leader_election.cc:288] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68aa9dace3cd4d479522e301c9245599 (127.16.98.130:34781), a8054e282ef845688c49c3d68b0fb708 (127.16.98.131:44647)
I20260807 22:45:54.913802 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.909328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:35707 (local address 127.16.98.130:34781)
0807 22:45:54.909421 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:54.909424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:54.909513 (+ 89us) server_negotiation.cc:407] Connection header received
0807 22:45:54.909600 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:54.909601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:54.909613 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:54.909676 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:54.910014 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.910883 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.912708 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.913322 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.913426 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.913439 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:54.913675 (+ 236us) server_negotiation.cc:299] Negotiation successful
0807 22:45:54.913723 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 22:45:54.913805 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.909450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:60789 (local address 127.16.98.131:44647)
0807 22:45:54.909503 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:54.909505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:54.909517 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:45:54.909600 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:54.909601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:54.909613 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:54.909676 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:54.910017 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.910848 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.912715 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.913398 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.913470 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.913478 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:54.913675 (+ 197us) server_negotiation.cc:299] Negotiation successful
0807 22:45:54.913717 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 22:45:54.913858 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.909202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:34781 (local address 127.16.98.129:35707)
0807 22:45:54.909428 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:54.909443 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:54.909521 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:54.909751 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:54.909753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:54.909757 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:54.909902 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.909905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.911065 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.911066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.912588 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.912593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.913466 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.913468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.913570 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.913578 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 22:45:54.913651 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 22:45:54.913724 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":98,"thread_start_us":69,"threads_started":1}
I20260807 22:45:54.913858 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.909204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:44647 (local address 127.16.98.129:60789)
0807 22:45:54.909430 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:54.909443 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:54.909517 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:54.909751 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:54.909753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:54.909756 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:54.909902 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.909907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.910978 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.910980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.912588 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:54.912593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.913491 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.913492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:54.913570 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.913578 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 22:45:54.913651 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 22:45:54.913724 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":100,"thread_start_us":71,"threads_started":1}
I20260807 22:45:54.914378 24821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172deaf9b84b46db987b2df619278a4c" candidate_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aa9dace3cd4d479522e301c9245599" is_pre_election: true
I20260807 22:45:54.914423 24966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172deaf9b84b46db987b2df619278a4c" candidate_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8054e282ef845688c49c3d68b0fb708" is_pre_election: true
I20260807 22:45:54.914577 24821 raft_consensus.cc:2454] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6034f1118ee42ccb46aee7eeb4cdde1 in term 0.
I20260807 22:45:54.914613 24966 raft_consensus.cc:2454] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6034f1118ee42ccb46aee7eeb4cdde1 in term 0.
I20260807 22:45:54.914856 24611 leader_election.cc:302] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6034f1118ee42ccb46aee7eeb4cdde1, a8054e282ef845688c49c3d68b0fb708; no voters:
I20260807 22:45:54.915005 25051 raft_consensus.cc:2781] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:54.915105 25051 raft_consensus.cc:482] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:54.915141 25051 raft_consensus.cc:3037] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:54.915938 25051 raft_consensus.cc:504] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:54.916126 25051 leader_election.cc:288] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [CANDIDATE]: Term 1 election: Requested vote from peers 68aa9dace3cd4d479522e301c9245599 (127.16.98.130:34781), a8054e282ef845688c49c3d68b0fb708 (127.16.98.131:44647)
I20260807 22:45:54.916316 24966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172deaf9b84b46db987b2df619278a4c" candidate_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8054e282ef845688c49c3d68b0fb708"
I20260807 22:45:54.916335 24821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172deaf9b84b46db987b2df619278a4c" candidate_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aa9dace3cd4d479522e301c9245599"
I20260807 22:45:54.916414 24821 raft_consensus.cc:3037] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:54.916414 24966 raft_consensus.cc:3037] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:54.917430 24966 raft_consensus.cc:2454] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6034f1118ee42ccb46aee7eeb4cdde1 in term 1.
I20260807 22:45:54.917430 24821 raft_consensus.cc:2454] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6034f1118ee42ccb46aee7eeb4cdde1 in term 1.
I20260807 22:45:54.917696 24610 leader_election.cc:302] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aa9dace3cd4d479522e301c9245599, a6034f1118ee42ccb46aee7eeb4cdde1; no voters:
I20260807 22:45:54.917821 25051 raft_consensus.cc:2781] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:54.917987 25051 raft_consensus.cc:686] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 LEADER]: Becoming Leader. State: Replica: a6034f1118ee42ccb46aee7eeb4cdde1, State: Running, Role: LEADER
I20260807 22:45:54.918097 25051 consensus_queue.cc:237] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:54.919448 24518 catalog_manager.cc:5495] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6034f1118ee42ccb46aee7eeb4cdde1 (127.16.98.129). New cstate: current_term: 1 leader_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20260807 22:45:54.936789 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:54.934047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.16.98.129:41199)
0807 22:45:54.934156 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:54.934159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:54.934189 (+ 30us) server_negotiation.cc:407] Connection header received
0807 22:45:54.934284 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:54.934285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:54.934309 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:54.934404 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:54.934595 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.935088 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.935876 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:54.936092 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:54.936180 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:54.936378 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:54.936502 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:54.936657 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:54.936676 (+ 19us) server_negotiation.cc:299] Negotiation successful
0807 22:45:54.936698 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 22:45:54.938895 24821 raft_consensus.cc:1261] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Refusing update from remote peer a6034f1118ee42ccb46aee7eeb4cdde1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:54.939339 24966 raft_consensus.cc:1261] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Refusing update from remote peer a6034f1118ee42ccb46aee7eeb4cdde1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:54.939409 25054 consensus_queue.cc:1035] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:54.939612 25051 consensus_queue.cc:1035] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:54.942054 25057 mvcc.cc:205] Tried to move back new op lower bound from 7316040724226584576 to 7316040724145209344. Current Snapshot: MvccSnapshot[applied={T|T < 7316040724226584576}]
I20260807 22:45:54.943187 25059 mvcc.cc:205] Tried to move back new op lower bound from 7316040724226584576 to 7316040724145209344. Current Snapshot: MvccSnapshot[applied={T|T < 7316040724226584576}]
I20260807 22:45:54.944758 24519 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:54.944867 24519 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:54.945683 24519 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd: Sending DeleteTablet for 3 replicas of tablet 172deaf9b84b46db987b2df619278a4c
I20260807 22:45:54.946019 24801 tablet_service.cc:1514] Processing DeleteTablet for tablet 172deaf9b84b46db987b2df619278a4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:54 UTC) from {username='oryx', principal='oryx/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:59656
I20260807 22:45:54.946249 24946 tablet_service.cc:1514] Processing DeleteTablet for tablet 172deaf9b84b46db987b2df619278a4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:54 UTC) from {username='oryx', principal='oryx/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:54794
I20260807 22:45:54.946377 24519 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:45:54.946548 24519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:54.946523 24658 tablet_service.cc:1514] Processing DeleteTablet for tablet 172deaf9b84b46db987b2df619278a4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:54 UTC) from {username='oryx', principal='oryx/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:48140
I20260807 22:45:54.946714 25068 tablet_replica.cc:330] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708: stopping tablet replica
I20260807 22:45:54.946869 25068 raft_consensus.cc:2229] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:54.946951 25067 tablet_replica.cc:330] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599: stopping tablet replica
I20260807 22:45:54.947106 25068 raft_consensus.cc:2258] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:54.947147 25069 tablet_replica.cc:330] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1: stopping tablet replica
I20260807 22:45:54.947255 25069 raft_consensus.cc:2229] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:45:54.947506 25069 raft_consensus.cc:2258] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:54.947788 25068 ts_tablet_manager.cc:1905] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:54.948011 25069 ts_tablet_manager.cc:1905] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:54.948629 24658 tablet_service.cc:1467] Processing CreateTablet for tablet 27a42e7c06ff4ce5b6110c9dc42c8e7b (DEFAULT_TABLE table=test-table [id=2e1a569e2a884ad29dd03a93daaea493]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:54.948774 24658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a42e7c06ff4ce5b6110c9dc42c8e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:54.950181 25068 ts_tablet_manager.cc:1918] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:54.950282 25049 tablet_bootstrap.cc:492] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1: Bootstrap starting.
I20260807 22:45:54.950345 25068 log.cc:1196] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/172deaf9b84b46db987b2df619278a4c
I20260807 22:45:54.950448 24801 tablet_service.cc:1467] Processing CreateTablet for tablet 27a42e7c06ff4ce5b6110c9dc42c8e7b (DEFAULT_TABLE table=test-table [id=2e1a569e2a884ad29dd03a93daaea493]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:54.950666 24801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a42e7c06ff4ce5b6110c9dc42c8e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:54.951014 25068 ts_tablet_manager.cc:1939] T 172deaf9b84b46db987b2df619278a4c P a8054e282ef845688c49c3d68b0fb708: Deleting consensus metadata
I20260807 22:45:54.951207 25049 tablet_bootstrap.cc:654] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:54.951570 25069 ts_tablet_manager.cc:1918] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:54.951647 25069 log.cc:1196] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/wal/wals/172deaf9b84b46db987b2df619278a4c
I20260807 22:45:54.951944 25069 ts_tablet_manager.cc:1939] T 172deaf9b84b46db987b2df619278a4c P a6034f1118ee42ccb46aee7eeb4cdde1: Deleting consensus metadata
I20260807 22:45:54.952543 25043 tablet_bootstrap.cc:492] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599: Bootstrap starting.
I20260807 22:45:54.952708 25049 tablet_bootstrap.cc:492] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1: No bootstrap required, opened a new log
I20260807 22:45:54.952757 25049 ts_tablet_manager.cc:1397] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260807 22:45:54.952886 24505 catalog_manager.cc:4841] TS a6034f1118ee42ccb46aee7eeb4cdde1 (127.16.98.129:41199): tablet 172deaf9b84b46db987b2df619278a4c (table test-table [id=96550447549e45aeaaaed0f783007db5]) successfully deleted
I20260807 22:45:54.952916 25049 raft_consensus.cc:348] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:54.952989 25049 raft_consensus.cc:374] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:54.953022 25049 raft_consensus.cc:729] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6034f1118ee42ccb46aee7eeb4cdde1, State: Initialized, Role: FOLLOWER
I20260807 22:45:54.953066 25049 consensus_queue.cc:260] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:54.953084 25043 tablet_bootstrap.cc:654] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:54.953156 25049 ts_tablet_manager.cc:1428] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260807 22:45:54.953213 24506 catalog_manager.cc:4841] TS a8054e282ef845688c49c3d68b0fb708 (127.16.98.131:44647): tablet 172deaf9b84b46db987b2df619278a4c (table test-table [id=96550447549e45aeaaaed0f783007db5]) successfully deleted
I20260807 22:45:54.953606 24946 tablet_service.cc:1467] Processing CreateTablet for tablet 27a42e7c06ff4ce5b6110c9dc42c8e7b (DEFAULT_TABLE table=test-table [id=2e1a569e2a884ad29dd03a93daaea493]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:54.953750 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a42e7c06ff4ce5b6110c9dc42c8e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:54.953799 25043 tablet_bootstrap.cc:492] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599: No bootstrap required, opened a new log
I20260807 22:45:54.953845 25043 ts_tablet_manager.cc:1397] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:54.954016 25043 raft_consensus.cc:348] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:54.954101 25043 raft_consensus.cc:374] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:54.954126 25043 raft_consensus.cc:729] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aa9dace3cd4d479522e301c9245599, State: Initialized, Role: FOLLOWER
I20260807 22:45:54.954176 25043 consensus_queue.cc:260] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:54.954262 25043 ts_tablet_manager.cc:1428] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:54.955066 25042 tablet_bootstrap.cc:492] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708: Bootstrap starting.
I20260807 22:45:54.955523 25042 tablet_bootstrap.cc:654] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:54.955917 25067 raft_consensus.cc:2229] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:54.956079 25067 raft_consensus.cc:2258] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:54.956357 25042 tablet_bootstrap.cc:492] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708: No bootstrap required, opened a new log
I20260807 22:45:54.956416 25042 ts_tablet_manager.cc:1397] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:54.956614 25042 raft_consensus.cc:348] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:54.956698 25042 raft_consensus.cc:374] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:54.956704 25067 ts_tablet_manager.cc:1905] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:54.956724 25042 raft_consensus.cc:729] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8054e282ef845688c49c3d68b0fb708, State: Initialized, Role: FOLLOWER
I20260807 22:45:54.956771 25042 consensus_queue.cc:260] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:54.956872 25042 ts_tablet_manager.cc:1428] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260807 22:45:54.958696 25067 ts_tablet_manager.cc:1918] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:54.958771 25067 log.cc:1196] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/172deaf9b84b46db987b2df619278a4c
I20260807 22:45:54.959041 25067 ts_tablet_manager.cc:1939] T 172deaf9b84b46db987b2df619278a4c P 68aa9dace3cd4d479522e301c9245599: Deleting consensus metadata
I20260807 22:45:54.959444 24505 catalog_manager.cc:4841] TS 68aa9dace3cd4d479522e301c9245599 (127.16.98.130:34781): tablet 172deaf9b84b46db987b2df619278a4c (table test-table [id=96550447549e45aeaaaed0f783007db5]) successfully deleted
W20260807 22:45:55.009228 25014 tablet.cc:2369] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:45:55.074692 25047 raft_consensus.cc:482] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:55.074858 25047 raft_consensus.cc:504] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.075376 25047 leader_election.cc:288] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6034f1118ee42ccb46aee7eeb4cdde1 (127.16.98.129:41199), 68aa9dace3cd4d479522e301c9245599 (127.16.98.130:34781)
I20260807 22:45:55.078990 24903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.075663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:41199 (local address 127.16.98.131:33127)
0807 22:45:55.075779 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:55.075795 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:55.075869 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:55.076157 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:55.076158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:55.076161 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:55.076271 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.076273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.077184 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.077185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.078070 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.078073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.078672 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.078673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.078771 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.078774 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:45:55.078852 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 22:45:55.078898 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260807 22:45:55.078989 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.075787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:33127 (local address 127.16.98.129:41199)
0807 22:45:55.075870 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:55.075873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:55.075884 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:45:55.075970 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:55.075971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:55.075984 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:55.076063 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:55.076367 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.077077 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.078183 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.078581 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.078673 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.078683 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:55.078858 (+ 175us) server_negotiation.cc:299] Negotiation successful
0807 22:45:55.078909 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 22:45:55.079059 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.075906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:33223 (local address 127.16.98.130:34781)
0807 22:45:55.075959 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:55.075961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:55.076378 (+ 417us) server_negotiation.cc:407] Connection header received
0807 22:45:55.076413 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:55.076414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:55.076425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:55.076492 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:55.076730 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.077322 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.078312 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.078680 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.078762 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.078771 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:55.078947 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 22:45:55.078986 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 22:45:55.079136 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.075811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:34781 (local address 127.16.98.131:33223)
0807 22:45:55.075991 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:55.076001 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:55.076110 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:55.076517 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:55.076519 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:55.076521 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:55.076651 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.076653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.077397 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.077398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.078212 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.078214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.078791 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.078792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.078865 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.078869 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:45:55.078935 (+ 66us) client_negotiation.cc:240] Negotiation successful
0807 22:45:55.078976 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
I20260807 22:45:55.079360 24821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a42e7c06ff4ce5b6110c9dc42c8e7b" candidate_uuid: "a8054e282ef845688c49c3d68b0fb708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aa9dace3cd4d479522e301c9245599" is_pre_election: true
I20260807 22:45:55.079453 24821 raft_consensus.cc:2454] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8054e282ef845688c49c3d68b0fb708 in term 0.
I20260807 22:45:55.079583 24678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a42e7c06ff4ce5b6110c9dc42c8e7b" candidate_uuid: "a8054e282ef845688c49c3d68b0fb708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" is_pre_election: true
I20260807 22:45:55.079679 24896 leader_election.cc:302] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aa9dace3cd4d479522e301c9245599, a8054e282ef845688c49c3d68b0fb708; no voters:
I20260807 22:45:55.079753 24678 raft_consensus.cc:2454] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8054e282ef845688c49c3d68b0fb708 in term 0.
I20260807 22:45:55.079842 25047 raft_consensus.cc:2781] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:55.079910 25047 raft_consensus.cc:482] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:55.079952 25047 raft_consensus.cc:3037] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:55.080720 25047 raft_consensus.cc:504] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.080892 25047 leader_election.cc:288] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [CANDIDATE]: Term 1 election: Requested vote from peers a6034f1118ee42ccb46aee7eeb4cdde1 (127.16.98.129:41199), 68aa9dace3cd4d479522e301c9245599 (127.16.98.130:34781)
I20260807 22:45:55.081130 24821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a42e7c06ff4ce5b6110c9dc42c8e7b" candidate_uuid: "a8054e282ef845688c49c3d68b0fb708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aa9dace3cd4d479522e301c9245599"
I20260807 22:45:55.081121 24678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a42e7c06ff4ce5b6110c9dc42c8e7b" candidate_uuid: "a8054e282ef845688c49c3d68b0fb708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1"
I20260807 22:45:55.081198 24821 raft_consensus.cc:3037] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:55.081218 24678 raft_consensus.cc:3037] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:55.081883 24821 raft_consensus.cc:2454] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8054e282ef845688c49c3d68b0fb708 in term 1.
I20260807 22:45:55.081892 24678 raft_consensus.cc:2454] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8054e282ef845688c49c3d68b0fb708 in term 1.
I20260807 22:45:55.082065 24896 leader_election.cc:302] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6034f1118ee42ccb46aee7eeb4cdde1, a8054e282ef845688c49c3d68b0fb708; no voters:
I20260807 22:45:55.082166 25047 raft_consensus.cc:2781] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:55.082412 25047 raft_consensus.cc:686] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [term 1 LEADER]: Becoming Leader. State: Replica: a8054e282ef845688c49c3d68b0fb708, State: Running, Role: LEADER
I20260807 22:45:55.082492 25047 consensus_queue.cc:237] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.083453 24518 catalog_manager.cc:5495] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8054e282ef845688c49c3d68b0fb708 (127.16.98.131). New cstate: current_term: 1 leader_uuid: "a8054e282ef845688c49c3d68b0fb708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } health_report { overall_health: HEALTHY } } }
Aug 07 22:45:55 dist-test-slave-rgb1 krb5kdc[24475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142754, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.190@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
I20260807 22:45:55.137926 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.129525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36419 (local address 127.0.0.1:54150)
0807 22:45:55.129750 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:55.129765 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:55.129835 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:55.130046 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:55.130047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:55.130353 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:55.130404 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.130405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.131272 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.131273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.132048 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.132051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.132408 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.132409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.132454 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.133192 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:55.133197 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:55.134873 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:55.136752 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:55.136760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:55.136762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:55.136979 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:55.137197 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:55.137198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:55.137198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:55.137273 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:55.137570 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:55.137578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:55.137624 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 22:45:55.137704 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 22:45:55.137811 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":168,"threads_started":1}
I20260807 22:45:55.137939 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.129610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.16.98.190:36419)
0807 22:45:55.129709 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:55.129711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:55.129892 (+ 181us) server_negotiation.cc:407] Connection header received
0807 22:45:55.129923 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:55.129924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:55.129938 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:55.130014 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:55.130552 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.131128 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.132146 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.132330 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.132397 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.135028 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:55.135029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:55.135030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:55.135037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:55.136675 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:55.137059 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:55.137059 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:55.137060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:55.137113 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:55.137365 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:55.137367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:55.137367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:55.137498 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:55.137568 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:55.137759 (+ 191us) server_negotiation.cc:299] Negotiation successful
0807 22:45:55.137864 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":70}
I20260807 22:45:55.140336 24518 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:54150:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:45:55.140540 24518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:55.142536 24946 tablet_service.cc:1467] Processing CreateTablet for tablet f51429e5de1e47a6b963de67188cbb77 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c0487d517d64e92bd1503b0b828b86d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:45:55.142536 24801 tablet_service.cc:1467] Processing CreateTablet for tablet f51429e5de1e47a6b963de67188cbb77 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c0487d517d64e92bd1503b0b828b86d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:45:55.142568 24658 tablet_service.cc:1467] Processing CreateTablet for tablet f51429e5de1e47a6b963de67188cbb77 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c0487d517d64e92bd1503b0b828b86d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:45:55.142725 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51429e5de1e47a6b963de67188cbb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:55.142728 24801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51429e5de1e47a6b963de67188cbb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:55.142728 24658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51429e5de1e47a6b963de67188cbb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:55.144176 25042 tablet_bootstrap.cc:492] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708: Bootstrap starting.
I20260807 22:45:55.144400 25043 tablet_bootstrap.cc:492] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599: Bootstrap starting.
I20260807 22:45:55.144656 25049 tablet_bootstrap.cc:492] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1: Bootstrap starting.
I20260807 22:45:55.144781 25042 tablet_bootstrap.cc:654] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:55.144822 25043 tablet_bootstrap.cc:654] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:55.145027 25049 tablet_bootstrap.cc:654] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:55.145603 25042 tablet_bootstrap.cc:492] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708: No bootstrap required, opened a new log
I20260807 22:45:55.145658 25042 ts_tablet_manager.cc:1397] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:55.145658 25043 tablet_bootstrap.cc:492] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599: No bootstrap required, opened a new log
I20260807 22:45:55.145711 25043 ts_tablet_manager.cc:1397] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:55.145820 25042 raft_consensus.cc:348] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.145900 25042 raft_consensus.cc:374] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:55.145915 25042 raft_consensus.cc:729] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8054e282ef845688c49c3d68b0fb708, State: Initialized, Role: FOLLOWER
I20260807 22:45:55.145877 25043 raft_consensus.cc:348] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.145978 25043 raft_consensus.cc:374] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:55.145979 25042 consensus_queue.cc:260] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.146011 25043 raft_consensus.cc:729] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aa9dace3cd4d479522e301c9245599, State: Initialized, Role: FOLLOWER
I20260807 22:45:55.146073 25042 ts_tablet_manager.cc:1428] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260807 22:45:55.146087 25043 consensus_queue.cc:260] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.146200 25043 ts_tablet_manager.cc:1428] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:55.146199 25047 tablet_replica.cc:439] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } } }
I20260807 22:45:55.146270 25047 tablet_replica.cc:442] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:55.146261 25046 tablet_replica.cc:439] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } } }
I20260807 22:45:55.146353 25046 tablet_replica.cc:442] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:55.146463 25049 tablet_bootstrap.cc:492] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1: No bootstrap required, opened a new log
I20260807 22:45:55.146508 25049 ts_tablet_manager.cc:1397] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260807 22:45:55.146628 25049 raft_consensus.cc:348] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.146709 25049 raft_consensus.cc:374] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:55.146730 25049 raft_consensus.cc:729] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6034f1118ee42ccb46aee7eeb4cdde1, State: Initialized, Role: FOLLOWER
I20260807 22:45:55.146778 25049 consensus_queue.cc:260] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.146883 25049 ts_tablet_manager.cc:1428] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:55.146924 25063 tablet_replica.cc:439] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } } }
I20260807 22:45:55.147029 25063 tablet_replica.cc:442] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:55.159315 25047 raft_consensus.cc:482] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:55.159487 25047 raft_consensus.cc:504] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.159659 25047 leader_election.cc:288] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6034f1118ee42ccb46aee7eeb4cdde1 (127.16.98.129:41199), 68aa9dace3cd4d479522e301c9245599 (127.16.98.130:34781)
I20260807 22:45:55.160032 24821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51429e5de1e47a6b963de67188cbb77" candidate_uuid: "a8054e282ef845688c49c3d68b0fb708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aa9dace3cd4d479522e301c9245599" is_pre_election: true
I20260807 22:45:55.160133 24821 raft_consensus.cc:2454] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8054e282ef845688c49c3d68b0fb708 in term 0.
I20260807 22:45:55.160045 24678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51429e5de1e47a6b963de67188cbb77" candidate_uuid: "a8054e282ef845688c49c3d68b0fb708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" is_pre_election: true
I20260807 22:45:55.160175 24678 raft_consensus.cc:2454] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8054e282ef845688c49c3d68b0fb708 in term 0.
I20260807 22:45:55.160300 24896 leader_election.cc:302] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aa9dace3cd4d479522e301c9245599, a8054e282ef845688c49c3d68b0fb708; no voters:
I20260807 22:45:55.160406 25047 raft_consensus.cc:2781] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:55.160465 25047 raft_consensus.cc:482] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:55.160516 25047 raft_consensus.cc:3037] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:55.161581 25047 raft_consensus.cc:504] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.161697 25047 leader_election.cc:288] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [CANDIDATE]: Term 1 election: Requested vote from peers a6034f1118ee42ccb46aee7eeb4cdde1 (127.16.98.129:41199), 68aa9dace3cd4d479522e301c9245599 (127.16.98.130:34781)
I20260807 22:45:55.161947 24678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51429e5de1e47a6b963de67188cbb77" candidate_uuid: "a8054e282ef845688c49c3d68b0fb708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1"
I20260807 22:45:55.161980 24821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51429e5de1e47a6b963de67188cbb77" candidate_uuid: "a8054e282ef845688c49c3d68b0fb708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aa9dace3cd4d479522e301c9245599"
I20260807 22:45:55.162065 24678 raft_consensus.cc:3037] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:55.162063 24821 raft_consensus.cc:3037] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:55.162792 24821 raft_consensus.cc:2454] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8054e282ef845688c49c3d68b0fb708 in term 1.
I20260807 22:45:55.162981 24896 leader_election.cc:302] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aa9dace3cd4d479522e301c9245599, a8054e282ef845688c49c3d68b0fb708; no voters:
I20260807 22:45:55.163156 25047 raft_consensus.cc:2781] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:55.163211 25047 raft_consensus.cc:686] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [term 1 LEADER]: Becoming Leader. State: Replica: a8054e282ef845688c49c3d68b0fb708, State: Running, Role: LEADER
I20260807 22:45:55.163206 24678 raft_consensus.cc:2454] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8054e282ef845688c49c3d68b0fb708 in term 1.
I20260807 22:45:55.163273 25047 consensus_queue.cc:237] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.163914 25074 tablet_replica.cc:439] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708: TxnStatusTablet state changed. Reason: New leader a8054e282ef845688c49c3d68b0fb708. Latest consensus state: current_term: 1 leader_uuid: "a8054e282ef845688c49c3d68b0fb708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } } }
I20260807 22:45:55.164022 25074 tablet_replica.cc:442] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:45:55.164237 25086 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:45:55.164359 25086 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:45:55.164636 24518 catalog_manager.cc:5495] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8054e282ef845688c49c3d68b0fb708 (127.16.98.131). New cstate: current_term: 1 leader_uuid: "a8054e282ef845688c49c3d68b0fb708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } health_report { overall_health: HEALTHY } } }
I20260807 22:45:55.172806 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.168842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.16.98.131:44647)
0807 22:45:55.168897 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:55.168899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:55.168985 (+ 86us) server_negotiation.cc:407] Connection header received
0807 22:45:55.169106 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:55.169107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:55.169130 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:55.169219 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:55.169368 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.169940 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.170821 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.171075 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.171185 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.172118 (+ 933us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:55.172238 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:55.172349 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:55.172684 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 22:45:55.172713 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 22:45:55.172834 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.168758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:44647 (local address 127.0.0.1:54810)
0807 22:45:55.168919 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:55.168931 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:55.169008 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:55.169225 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:55.169226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:55.169227 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:45:55.169286 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.169288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.170051 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.170053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.170721 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.170723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.171893 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.171894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.171956 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.171982 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:45:55.172342 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:45:55.172354 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 22:45:55.172671 (+ 317us) client_negotiation.cc:240] Negotiation successful
0807 22:45:55.172742 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260807 22:45:55.175072 24821 raft_consensus.cc:1261] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Refusing update from remote peer a8054e282ef845688c49c3d68b0fb708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:55.175539 24678 raft_consensus.cc:1261] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 FOLLOWER]: Refusing update from remote peer a8054e282ef845688c49c3d68b0fb708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:55.175784 25074 consensus_queue.cc:1035] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:55.176100 25047 consensus_queue.cc:1035] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:55.176798 25046 tablet_replica.cc:439] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599: TxnStatusTablet state changed. Reason: New leader a8054e282ef845688c49c3d68b0fb708. Latest consensus state: current_term: 1 leader_uuid: "a8054e282ef845688c49c3d68b0fb708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } } }
I20260807 22:45:55.176946 25046 tablet_replica.cc:442] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:55.177047 25063 tablet_replica.cc:439] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1: TxnStatusTablet state changed. Reason: New leader a8054e282ef845688c49c3d68b0fb708. Latest consensus state: current_term: 1 leader_uuid: "a8054e282ef845688c49c3d68b0fb708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } } }
I20260807 22:45:55.177145 25063 tablet_replica.cc:442] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:55.177629 25058 mvcc.cc:205] Tried to move back new op lower bound from 7316040725194055680 to 7316040725149093888. Current Snapshot: MvccSnapshot[applied={T|T < 7316040725194055680}]
I20260807 22:45:55.178184 25046 tablet_replica.cc:439] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8054e282ef845688c49c3d68b0fb708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } } }
I20260807 22:45:55.178303 25046 tablet_replica.cc:442] T f51429e5de1e47a6b963de67188cbb77 P 68aa9dace3cd4d479522e301c9245599: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:55.179030 25047 tablet_replica.cc:439] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8054e282ef845688c49c3d68b0fb708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } } }
I20260807 22:45:55.179133 25047 tablet_replica.cc:442] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:45:55.179204 25074 tablet_replica.cc:439] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8054e282ef845688c49c3d68b0fb708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } } }
I20260807 22:45:55.179307 25074 tablet_replica.cc:442] T f51429e5de1e47a6b963de67188cbb77 P a8054e282ef845688c49c3d68b0fb708: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:45:55.179782 25063 tablet_replica.cc:439] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8054e282ef845688c49c3d68b0fb708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } } }
I20260807 22:45:55.179883 25063 tablet_replica.cc:442] T f51429e5de1e47a6b963de67188cbb77 P a6034f1118ee42ccb46aee7eeb4cdde1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:45:55.183090 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.179620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.16.98.131:44647)
0807 22:45:55.179661 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:55.179663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:55.179671 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:45:55.179698 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:55.179699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:55.179710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:55.179776 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:55.179997 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.181016 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.181989 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.182331 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.182416 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.182594 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:55.182634 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:55.182714 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:55.182977 (+ 263us) server_negotiation.cc:299] Negotiation successful
0807 22:45:55.183005 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260807 22:45:55.188345 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.186008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:38071 (local address 127.16.98.131:44647)
0807 22:45:55.186111 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:55.186113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:55.186128 (+ 15us) server_negotiation.cc:407] Connection header received
0807 22:45:55.186171 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:55.186172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:55.186181 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:55.186229 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:55.186356 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.186825 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.187574 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.187769 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.187843 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.188034 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:55.188064 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:55.188133 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:55.188233 (+ 100us) server_negotiation.cc:299] Negotiation successful
0807 22:45:55.188260 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 22:45:55.188345 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.185961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:44647 (local address 127.16.98.131:38071)
0807 22:45:55.186059 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:55.186073 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:55.186115 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:55.186233 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:55.186234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:55.186235 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:45:55.186296 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.186298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.186909 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.186910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.187468 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.187469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.187863 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.187864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.187922 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.187936 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:45:55.188134 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:45:55.188135 (+ 1us) client_negotiation.cc:769] Sending connection context
0807 22:45:55.188230 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 22:45:55.188275 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":2}
I20260807 22:45:55.192394 24678 raft_consensus.cc:1261] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 FOLLOWER]: Refusing update from remote peer a8054e282ef845688c49c3d68b0fb708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:55.192446 24821 raft_consensus.cc:1261] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Refusing update from remote peer a8054e282ef845688c49c3d68b0fb708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:55.192690 25074 consensus_queue.cc:1035] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:55.192807 25047 consensus_queue.cc:1035] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:55.209020 24518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:55.209153 24518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:55.209860 24518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd: Sending DeleteTablet for 3 replicas of tablet 27a42e7c06ff4ce5b6110c9dc42c8e7b
I20260807 22:45:55.210141 24658 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a42e7c06ff4ce5b6110c9dc42c8e7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:55 UTC) from {username='oryx', principal='oryx/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:48140
I20260807 22:45:55.210217 24945 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a42e7c06ff4ce5b6110c9dc42c8e7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:55 UTC) from {username='oryx', principal='oryx/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:54794
I20260807 22:45:55.210280 25069 tablet_replica.cc:330] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1: stopping tablet replica
I20260807 22:45:55.210290 24801 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a42e7c06ff4ce5b6110c9dc42c8e7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:55 UTC) from {username='oryx', principal='oryx/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:59656
I20260807 22:45:55.210318 25068 tablet_replica.cc:330] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708: stopping tablet replica
I20260807 22:45:55.210359 25069 raft_consensus.cc:2229] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:55.210369 25068 raft_consensus.cc:2229] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:45:55.210388 25067 tablet_replica.cc:330] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599: stopping tablet replica
I20260807 22:45:55.210407 25069 raft_consensus.cc:2258] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:55.210441 25067 raft_consensus.cc:2229] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:55.210493 25068 raft_consensus.cc:2258] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:55.210500 25067 raft_consensus.cc:2258] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:55.211090 25067 ts_tablet_manager.cc:1905] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:55.212990 25067 ts_tablet_manager.cc:1918] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:45:55.213073 25067 log.cc:1196] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/27a42e7c06ff4ce5b6110c9dc42c8e7b
I20260807 22:45:55.213189 25068 ts_tablet_manager.cc:1905] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:55.213327 25067 ts_tablet_manager.cc:1939] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P 68aa9dace3cd4d479522e301c9245599: Deleting consensus metadata
I20260807 22:45:55.213822 24505 catalog_manager.cc:4841] TS 68aa9dace3cd4d479522e301c9245599 (127.16.98.130:34781): tablet 27a42e7c06ff4ce5b6110c9dc42c8e7b (table test-table [id=2e1a569e2a884ad29dd03a93daaea493]) successfully deleted
I20260807 22:45:55.215061 25068 ts_tablet_manager.cc:1918] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:45:55.215143 25068 log.cc:1196] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/27a42e7c06ff4ce5b6110c9dc42c8e7b
I20260807 22:45:55.215395 25068 ts_tablet_manager.cc:1939] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a8054e282ef845688c49c3d68b0fb708: Deleting consensus metadata
I20260807 22:45:55.215973 24506 catalog_manager.cc:4841] TS a8054e282ef845688c49c3d68b0fb708 (127.16.98.131:44647): tablet 27a42e7c06ff4ce5b6110c9dc42c8e7b (table test-table [id=2e1a569e2a884ad29dd03a93daaea493]) successfully deleted
I20260807 22:45:55.220899 25069 ts_tablet_manager.cc:1905] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:55.222741 25069 ts_tablet_manager.cc:1918] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:45:55.222827 25069 log.cc:1196] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/wal/wals/27a42e7c06ff4ce5b6110c9dc42c8e7b
I20260807 22:45:55.223101 25069 ts_tablet_manager.cc:1939] T 27a42e7c06ff4ce5b6110c9dc42c8e7b P a6034f1118ee42ccb46aee7eeb4cdde1: Deleting consensus metadata
I20260807 22:45:55.223572 24505 catalog_manager.cc:4841] TS a6034f1118ee42ccb46aee7eeb4cdde1 (127.16.98.129:41199): tablet 27a42e7c06ff4ce5b6110c9dc42c8e7b (table test-table [id=2e1a569e2a884ad29dd03a93daaea493]) successfully deleted
Aug 07 22:45:55 dist-test-slave-rgb1 krb5kdc[24475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 22:45:55 dist-test-slave-rgb1 krb5kdc[24475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:45:55 dist-test-slave-rgb1 krb5kdc[24475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
W20260807 22:45:55.241219 24570 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:55.234180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.16.98.190:36419)
0807 22:45:55.234333 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:55.234336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:55.234351 (+ 15us) server_negotiation.cc:407] Connection header received
0807 22:45:55.234391 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:55.234392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:55.234404 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:55.234475 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:55.235332 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.235931 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.237187 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.237411 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.237493 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.241121 (+ 3628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54166: BlockingRecv error: recv got EOF from 127.0.0.1:54166 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Aug 07 22:45:55 dist-test-slave-rgb1 krb5kdc[24475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM
I20260807 22:45:55.250774 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.242222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.16.98.190:36419)
0807 22:45:55.242288 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:55.242291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:55.242418 (+ 127us) server_negotiation.cc:407] Connection header received
0807 22:45:55.242555 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:55.242557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:55.242569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:55.242620 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:55.243181 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.243730 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.244619 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.244847 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.244951 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.247624 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:55.247626 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:55.247627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:55.247636 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:55.249455 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:55.249866 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:55.249868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:55.249868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:55.249938 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:55.250171 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:55.250172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:55.250173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:55.250319 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:55.250444 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:55.250557 (+ 113us) server_negotiation.cc:299] Negotiation successful
0807 22:45:55.250649 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 22:45:55.252527 24518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:45:55.252797 24518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:55.254616 24945 tablet_service.cc:1467] Processing CreateTablet for tablet 6cbb33544cbd42018bb39aeb8f7e71e6 (DEFAULT_TABLE table=test-table [id=90a9e3f09c584fcbb39e51a02cd1d145]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:55.254616 24658 tablet_service.cc:1467] Processing CreateTablet for tablet 6cbb33544cbd42018bb39aeb8f7e71e6 (DEFAULT_TABLE table=test-table [id=90a9e3f09c584fcbb39e51a02cd1d145]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:55.254688 24801 tablet_service.cc:1467] Processing CreateTablet for tablet 6cbb33544cbd42018bb39aeb8f7e71e6 (DEFAULT_TABLE table=test-table [id=90a9e3f09c584fcbb39e51a02cd1d145]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:55.254786 24658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cbb33544cbd42018bb39aeb8f7e71e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:55.254783 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cbb33544cbd42018bb39aeb8f7e71e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:55.254827 24801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cbb33544cbd42018bb39aeb8f7e71e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:55.256289 25042 tablet_bootstrap.cc:492] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708: Bootstrap starting.
I20260807 22:45:55.256325 25049 tablet_bootstrap.cc:492] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1: Bootstrap starting.
I20260807 22:45:55.256673 25042 tablet_bootstrap.cc:654] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:55.256712 25049 tablet_bootstrap.cc:654] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:55.256996 25043 tablet_bootstrap.cc:492] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599: Bootstrap starting.
I20260807 22:45:55.257422 25043 tablet_bootstrap.cc:654] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:55.257486 25042 tablet_bootstrap.cc:492] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708: No bootstrap required, opened a new log
I20260807 22:45:55.257546 25042 ts_tablet_manager.cc:1397] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:55.257773 25042 raft_consensus.cc:348] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.257879 25042 raft_consensus.cc:374] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:55.257922 25042 raft_consensus.cc:729] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8054e282ef845688c49c3d68b0fb708, State: Initialized, Role: FOLLOWER
I20260807 22:45:55.257972 25042 consensus_queue.cc:260] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.258085 25042 ts_tablet_manager.cc:1428] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:55.258147 25049 tablet_bootstrap.cc:492] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1: No bootstrap required, opened a new log
I20260807 22:45:55.258205 25049 ts_tablet_manager.cc:1397] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260807 22:45:55.258322 25049 raft_consensus.cc:348] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.258397 25049 raft_consensus.cc:374] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:55.258420 25049 raft_consensus.cc:729] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6034f1118ee42ccb46aee7eeb4cdde1, State: Initialized, Role: FOLLOWER
I20260807 22:45:55.258467 25049 consensus_queue.cc:260] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.258503 25043 tablet_bootstrap.cc:492] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599: No bootstrap required, opened a new log
I20260807 22:45:55.258560 25049 ts_tablet_manager.cc:1428] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:55.258559 25043 ts_tablet_manager.cc:1397] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260807 22:45:55.258736 25043 raft_consensus.cc:348] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.258853 25043 raft_consensus.cc:374] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:55.258888 25043 raft_consensus.cc:729] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aa9dace3cd4d479522e301c9245599, State: Initialized, Role: FOLLOWER
I20260807 22:45:55.258966 25043 consensus_queue.cc:260] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.259078 25043 ts_tablet_manager.cc:1428] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260807 22:45:55.407353 25091 raft_consensus.cc:482] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:55.407467 25091 raft_consensus.cc:504] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.407631 25091 leader_election.cc:288] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6034f1118ee42ccb46aee7eeb4cdde1 (127.16.98.129:41199), 68aa9dace3cd4d479522e301c9245599 (127.16.98.130:34781)
I20260807 22:45:55.407908 24678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cbb33544cbd42018bb39aeb8f7e71e6" candidate_uuid: "a8054e282ef845688c49c3d68b0fb708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" is_pre_election: true
I20260807 22:45:55.408017 24678 raft_consensus.cc:2454] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8054e282ef845688c49c3d68b0fb708 in term 0.
I20260807 22:45:55.408123 24821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cbb33544cbd42018bb39aeb8f7e71e6" candidate_uuid: "a8054e282ef845688c49c3d68b0fb708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aa9dace3cd4d479522e301c9245599" is_pre_election: true
I20260807 22:45:55.408218 24821 raft_consensus.cc:2454] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8054e282ef845688c49c3d68b0fb708 in term 0.
I20260807 22:45:55.408435 24896 leader_election.cc:302] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6034f1118ee42ccb46aee7eeb4cdde1, a8054e282ef845688c49c3d68b0fb708; no voters:
I20260807 22:45:55.408622 25091 raft_consensus.cc:2781] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:55.408665 25091 raft_consensus.cc:482] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:55.408685 25091 raft_consensus.cc:3037] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:55.409634 25091 raft_consensus.cc:504] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.409777 25091 leader_election.cc:288] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [CANDIDATE]: Term 1 election: Requested vote from peers a6034f1118ee42ccb46aee7eeb4cdde1 (127.16.98.129:41199), 68aa9dace3cd4d479522e301c9245599 (127.16.98.130:34781)
I20260807 22:45:55.409924 24678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cbb33544cbd42018bb39aeb8f7e71e6" candidate_uuid: "a8054e282ef845688c49c3d68b0fb708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1"
I20260807 22:45:55.410003 24678 raft_consensus.cc:3037] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:55.410967 24678 raft_consensus.cc:2454] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8054e282ef845688c49c3d68b0fb708 in term 1.
I20260807 22:45:55.411095 24821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cbb33544cbd42018bb39aeb8f7e71e6" candidate_uuid: "a8054e282ef845688c49c3d68b0fb708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68aa9dace3cd4d479522e301c9245599"
I20260807 22:45:55.411185 24821 raft_consensus.cc:3037] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:55.411420 24896 leader_election.cc:302] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6034f1118ee42ccb46aee7eeb4cdde1, a8054e282ef845688c49c3d68b0fb708; no voters:
I20260807 22:45:55.411608 25091 raft_consensus.cc:2781] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:55.411705 25091 raft_consensus.cc:686] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [term 1 LEADER]: Becoming Leader. State: Replica: a8054e282ef845688c49c3d68b0fb708, State: Running, Role: LEADER
I20260807 22:45:55.411870 25091 consensus_queue.cc:237] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.412012 24821 raft_consensus.cc:2454] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8054e282ef845688c49c3d68b0fb708 in term 1.
I20260807 22:45:55.412487 24519 catalog_manager.cc:5495] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8054e282ef845688c49c3d68b0fb708 (127.16.98.131). New cstate: current_term: 1 leader_uuid: "a8054e282ef845688c49c3d68b0fb708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } health_report { overall_health: HEALTHY } } }
I20260807 22:45:55.450289 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.447810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.16.98.131:44647)
0807 22:45:55.447886 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:55.447888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:55.447900 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:45:55.447947 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:55.447948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:55.447957 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:55.448068 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:55.448237 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.448790 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.449642 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.449821 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.449876 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.450064 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:55.450092 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:55.450132 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:55.450209 (+ 77us) server_negotiation.cc:299] Negotiation successful
0807 22:45:55.450227 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 22:45:55.451459 24821 raft_consensus.cc:1261] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Refusing update from remote peer a8054e282ef845688c49c3d68b0fb708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:55.451470 24678 raft_consensus.cc:1261] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 FOLLOWER]: Refusing update from remote peer a8054e282ef845688c49c3d68b0fb708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:55.451761 25047 consensus_queue.cc:1035] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:55.451907 25091 consensus_queue.cc:1035] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:55.454474 24518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:55.454598 24518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:55.455487 24518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd: Sending DeleteTablet for 3 replicas of tablet 6cbb33544cbd42018bb39aeb8f7e71e6
I20260807 22:45:55.455890 24658 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cbb33544cbd42018bb39aeb8f7e71e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:55 UTC) from {username='oryx', principal='oryx/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:48140
I20260807 22:45:55.455936 24801 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cbb33544cbd42018bb39aeb8f7e71e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:55 UTC) from {username='oryx', principal='oryx/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:59656
I20260807 22:45:55.455965 24945 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cbb33544cbd42018bb39aeb8f7e71e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:55 UTC) from {username='oryx', principal='oryx/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:54794
I20260807 22:45:55.456005 25069 tablet_replica.cc:330] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1: stopping tablet replica
I20260807 22:45:55.456043 25067 tablet_replica.cc:330] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599: stopping tablet replica
I20260807 22:45:55.456058 25068 tablet_replica.cc:330] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708: stopping tablet replica
I20260807 22:45:55.456089 25067 raft_consensus.cc:2229] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:55.456105 25068 raft_consensus.cc:2229] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:45:55.456130 25069 raft_consensus.cc:2229] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:55.456140 25067 raft_consensus.cc:2258] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:55.456166 25068 raft_consensus.cc:2258] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:55.456187 25069 raft_consensus.cc:2258] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:55.456525 25068 ts_tablet_manager.cc:1905] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:55.456540 25067 ts_tablet_manager.cc:1905] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:55.456712 25069 ts_tablet_manager.cc:1905] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:55.456658 24518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:45:55.456802 24518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:45:55.458060 24658 tablet_service.cc:1467] Processing CreateTablet for tablet 7737780ed1da4ccc8802a47411403301 (DEFAULT_TABLE table=test-table [id=4a14520c8ebe4adcbb0775b18326eb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:55.458168 24658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7737780ed1da4ccc8802a47411403301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:55.458457 24945 tablet_service.cc:1467] Processing CreateTablet for tablet 7737780ed1da4ccc8802a47411403301 (DEFAULT_TABLE table=test-table [id=4a14520c8ebe4adcbb0775b18326eb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:55.458483 24801 tablet_service.cc:1467] Processing CreateTablet for tablet 7737780ed1da4ccc8802a47411403301 (DEFAULT_TABLE table=test-table [id=4a14520c8ebe4adcbb0775b18326eb72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:45:55.458468 25068 ts_tablet_manager.cc:1918] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:55.458516 25067 ts_tablet_manager.cc:1918] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:55.458559 25067 log.cc:1196] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/6cbb33544cbd42018bb39aeb8f7e71e6
I20260807 22:45:55.458559 25068 log.cc:1196] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/6cbb33544cbd42018bb39aeb8f7e71e6
I20260807 22:45:55.458578 24801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7737780ed1da4ccc8802a47411403301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:55.458602 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7737780ed1da4ccc8802a47411403301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:55.458796 25067 ts_tablet_manager.cc:1939] T 6cbb33544cbd42018bb39aeb8f7e71e6 P 68aa9dace3cd4d479522e301c9245599: Deleting consensus metadata
I20260807 22:45:55.458796 25068 ts_tablet_manager.cc:1939] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a8054e282ef845688c49c3d68b0fb708: Deleting consensus metadata
I20260807 22:45:55.459432 24506 catalog_manager.cc:4841] TS a8054e282ef845688c49c3d68b0fb708 (127.16.98.131:44647): tablet 6cbb33544cbd42018bb39aeb8f7e71e6 (table test-table [id=90a9e3f09c584fcbb39e51a02cd1d145]) successfully deleted
I20260807 22:45:55.459784 25043 tablet_bootstrap.cc:492] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599: Bootstrap starting.
I20260807 22:45:55.459879 25042 tablet_bootstrap.cc:492] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708: Bootstrap starting.
I20260807 22:45:55.459908 24505 catalog_manager.cc:4841] TS 68aa9dace3cd4d479522e301c9245599 (127.16.98.130:34781): tablet 6cbb33544cbd42018bb39aeb8f7e71e6 (table test-table [id=90a9e3f09c584fcbb39e51a02cd1d145]) successfully deleted
I20260807 22:45:55.459961 25069 ts_tablet_manager.cc:1918] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:45:55.460036 25069 log.cc:1196] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-0/wal/wals/6cbb33544cbd42018bb39aeb8f7e71e6
I20260807 22:45:55.460220 25042 tablet_bootstrap.cc:654] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:55.460266 25069 ts_tablet_manager.cc:1939] T 6cbb33544cbd42018bb39aeb8f7e71e6 P a6034f1118ee42ccb46aee7eeb4cdde1: Deleting consensus metadata
I20260807 22:45:55.460445 25043 tablet_bootstrap.cc:654] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:55.460939 25042 tablet_bootstrap.cc:492] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708: No bootstrap required, opened a new log
I20260807 22:45:55.460983 25042 ts_tablet_manager.cc:1397] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:55.461115 25042 raft_consensus.cc:348] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.461190 25042 raft_consensus.cc:374] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:55.461205 25042 raft_consensus.cc:729] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8054e282ef845688c49c3d68b0fb708, State: Initialized, Role: FOLLOWER
I20260807 22:45:55.461259 25042 consensus_queue.cc:260] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.461328 25049 tablet_bootstrap.cc:492] T 7737780ed1da4ccc8802a47411403301 P a6034f1118ee42ccb46aee7eeb4cdde1: Bootstrap starting.
I20260807 22:45:55.461359 25042 ts_tablet_manager.cc:1428] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:55.461436 24505 catalog_manager.cc:4841] TS a6034f1118ee42ccb46aee7eeb4cdde1 (127.16.98.129:41199): tablet 6cbb33544cbd42018bb39aeb8f7e71e6 (table test-table [id=90a9e3f09c584fcbb39e51a02cd1d145]) successfully deleted
I20260807 22:45:55.461714 25049 tablet_bootstrap.cc:654] T 7737780ed1da4ccc8802a47411403301 P a6034f1118ee42ccb46aee7eeb4cdde1: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:55.462412 25049 tablet_bootstrap.cc:492] T 7737780ed1da4ccc8802a47411403301 P a6034f1118ee42ccb46aee7eeb4cdde1: No bootstrap required, opened a new log
I20260807 22:45:55.462483 25043 tablet_bootstrap.cc:492] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599: No bootstrap required, opened a new log
I20260807 22:45:55.462471 25049 ts_tablet_manager.cc:1397] T 7737780ed1da4ccc8802a47411403301 P a6034f1118ee42ccb46aee7eeb4cdde1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:55.462538 25043 ts_tablet_manager.cc:1397] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260807 22:45:55.462604 25049 raft_consensus.cc:348] T 7737780ed1da4ccc8802a47411403301 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.462685 25049 raft_consensus.cc:374] T 7737780ed1da4ccc8802a47411403301 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:55.462700 25049 raft_consensus.cc:729] T 7737780ed1da4ccc8802a47411403301 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6034f1118ee42ccb46aee7eeb4cdde1, State: Initialized, Role: FOLLOWER
I20260807 22:45:55.462711 25043 raft_consensus.cc:348] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.462755 25049 consensus_queue.cc:260] T 7737780ed1da4ccc8802a47411403301 P a6034f1118ee42ccb46aee7eeb4cdde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.462790 25043 raft_consensus.cc:374] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:55.462823 25043 raft_consensus.cc:729] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aa9dace3cd4d479522e301c9245599, State: Initialized, Role: FOLLOWER
I20260807 22:45:55.462844 25049 ts_tablet_manager.cc:1428] T 7737780ed1da4ccc8802a47411403301 P a6034f1118ee42ccb46aee7eeb4cdde1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:55.462877 25043 consensus_queue.cc:260] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.462976 25043 ts_tablet_manager.cc:1428] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:55.507266 25046 raft_consensus.cc:482] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:45:55.507375 25046 raft_consensus.cc:504] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.507827 25046 leader_election.cc:288] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6034f1118ee42ccb46aee7eeb4cdde1 (127.16.98.129:41199), a8054e282ef845688c49c3d68b0fb708 (127.16.98.131:44647)
I20260807 22:45:55.511765 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.508073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:41199 (local address 127.16.98.130:53083)
0807 22:45:55.508234 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:55.508247 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:55.508320 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:55.508484 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:55.508485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:55.508488 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:55.508631 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.508634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.509756 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.509759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.510838 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.510842 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.511483 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.511485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.511560 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.511566 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 22:45:55.511638 (+ 72us) client_negotiation.cc:240] Negotiation successful
0807 22:45:55.511677 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":90,"thread_start_us":80,"threads_started":1}
I20260807 22:45:55.511888 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.508183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:53083 (local address 127.16.98.129:41199)
0807 22:45:55.508230 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:55.508232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:55.508337 (+ 105us) server_negotiation.cc:407] Connection header received
0807 22:45:55.508373 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:55.508374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:55.508384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:55.508465 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:55.508809 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.509612 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.510952 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.511344 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.511409 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.511417 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:55.511700 (+ 283us) server_negotiation.cc:299] Negotiation successful
0807 22:45:55.511794 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 22:45:55.511991 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.508169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:32899 (local address 127.16.98.131:44647)
0807 22:45:55.508227 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:55.508230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:55.508330 (+ 100us) server_negotiation.cc:407] Connection header received
0807 22:45:55.508373 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:55.508374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:55.508384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:55.508465 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:45:55.508810 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.509600 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.510988 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.511444 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.511540 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.511557 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:55.511854 (+ 297us) server_negotiation.cc:299] Negotiation successful
0807 22:45:55.511907 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 22:45:55.512260 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.508073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:44647 (local address 127.16.98.130:32899)
0807 22:45:55.508234 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:55.508247 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:55.508320 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:55.508483 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:55.508484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:55.508486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:45:55.508633 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.508634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.509769 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.509771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.510836 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.510840 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.511589 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.511590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.511735 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.511742 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 22:45:55.511823 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 22:45:55.511985 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":123,"thread_start_us":80,"threads_started":1}
I20260807 22:45:55.512269 24678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7737780ed1da4ccc8802a47411403301" candidate_uuid: "68aa9dace3cd4d479522e301c9245599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" is_pre_election: true
I20260807 22:45:55.512418 24678 raft_consensus.cc:2454] T 7737780ed1da4ccc8802a47411403301 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aa9dace3cd4d479522e301c9245599 in term 0.
I20260807 22:45:55.512598 24966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7737780ed1da4ccc8802a47411403301" candidate_uuid: "68aa9dace3cd4d479522e301c9245599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8054e282ef845688c49c3d68b0fb708" is_pre_election: true
I20260807 22:45:55.512708 24966 raft_consensus.cc:2454] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aa9dace3cd4d479522e301c9245599 in term 0.
I20260807 22:45:55.512665 24750 leader_election.cc:302] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aa9dace3cd4d479522e301c9245599, a6034f1118ee42ccb46aee7eeb4cdde1; no voters:
I20260807 22:45:55.512861 25046 raft_consensus.cc:2781] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:45:55.513017 25046 raft_consensus.cc:482] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:45:55.513057 25046 raft_consensus.cc:3037] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:55.513797 25046 raft_consensus.cc:504] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.513967 25046 leader_election.cc:288] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [CANDIDATE]: Term 1 election: Requested vote from peers a6034f1118ee42ccb46aee7eeb4cdde1 (127.16.98.129:41199), a8054e282ef845688c49c3d68b0fb708 (127.16.98.131:44647)
I20260807 22:45:55.514120 24966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7737780ed1da4ccc8802a47411403301" candidate_uuid: "68aa9dace3cd4d479522e301c9245599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8054e282ef845688c49c3d68b0fb708"
I20260807 22:45:55.514220 24966 raft_consensus.cc:3037] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:55.514258 24678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7737780ed1da4ccc8802a47411403301" candidate_uuid: "68aa9dace3cd4d479522e301c9245599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1"
I20260807 22:45:55.514335 24678 raft_consensus.cc:3037] T 7737780ed1da4ccc8802a47411403301 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:45:55.514828 24966 raft_consensus.cc:2454] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aa9dace3cd4d479522e301c9245599 in term 1.
I20260807 22:45:55.515038 24751 leader_election.cc:302] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aa9dace3cd4d479522e301c9245599, a8054e282ef845688c49c3d68b0fb708; no voters:
I20260807 22:45:55.515043 24678 raft_consensus.cc:2454] T 7737780ed1da4ccc8802a47411403301 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aa9dace3cd4d479522e301c9245599 in term 1.
I20260807 22:45:55.515224 25046 raft_consensus.cc:2781] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:55.515411 25046 raft_consensus.cc:686] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [term 1 LEADER]: Becoming Leader. State: Replica: 68aa9dace3cd4d479522e301c9245599, State: Running, Role: LEADER
I20260807 22:45:55.515497 25046 consensus_queue.cc:237] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } }
I20260807 22:45:55.516347 24518 catalog_manager.cc:5495] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68aa9dace3cd4d479522e301c9245599 (127.16.98.130). New cstate: current_term: 1 leader_uuid: "68aa9dace3cd4d479522e301c9245599" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68aa9dace3cd4d479522e301c9245599" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20260807 22:45:55.528964 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.526307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59660 (local address 127.16.98.130:34781)
0807 22:45:55.526470 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:55.526473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:55.526483 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:45:55.526573 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:55.526574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:55.526603 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:55.526688 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:55.526805 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.527319 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.528096 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.528293 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.528382 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.528600 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:55.528718 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:55.528808 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:55.528851 (+ 43us) server_negotiation.cc:299] Negotiation successful
0807 22:45:55.528875 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260807 22:45:55.535022 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.532154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:36021 (local address 127.16.98.131:44647)
0807 22:45:55.532242 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:55.532244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:55.532256 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:45:55.532299 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:55.532300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:55.532312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:55.532376 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:55.532520 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.533166 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.534088 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.534304 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.534365 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.534608 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:55.534637 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:55.534788 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:55.534936 (+ 148us) server_negotiation.cc:299] Negotiation successful
0807 22:45:55.534964 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 22:45:55.535072 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.531939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:44647 (local address 127.16.98.130:36021)
0807 22:45:55.532174 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:55.532187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:55.532241 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:55.532385 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:55.532386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:55.532387 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:45:55.532440 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.532441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.533255 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.533256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.533968 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.533972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.534405 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.534406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.534511 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.534528 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:45:55.534829 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:45:55.534832 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:45:55.534901 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 22:45:55.534956 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":122,"threads_started":1}
I20260807 22:45:55.537914 24678 raft_consensus.cc:1261] T 7737780ed1da4ccc8802a47411403301 P a6034f1118ee42ccb46aee7eeb4cdde1 [term 1 FOLLOWER]: Refusing update from remote peer 68aa9dace3cd4d479522e301c9245599: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:55.537930 24966 raft_consensus.cc:1261] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Refusing update from remote peer 68aa9dace3cd4d479522e301c9245599: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:45:55.538174 25046 consensus_queue.cc:1035] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6034f1118ee42ccb46aee7eeb4cdde1" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:55.538309 25121 consensus_queue.cc:1035] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8054e282ef845688c49c3d68b0fb708" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:45:55.546761 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.544148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:44929 (local address 127.16.98.130:34781)
0807 22:45:55.544194 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:55.544196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:55.544204 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:45:55.544269 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:55.544270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:55.544281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:55.544346 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:45:55.544485 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.545060 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.546083 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.546359 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.546414 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.546584 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:45:55.546613 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:45:55.546652 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:55.546685 (+ 33us) server_negotiation.cc:299] Negotiation successful
0807 22:45:55.546702 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 22:45:55.546783 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.544092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:34781 (local address 127.16.98.131:44929)
0807 22:45:55.544146 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:55.544156 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:55.544195 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:55.544348 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:55.544348 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:55.544349 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 22:45:55.544394 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.544403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.545156 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.545157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.545985 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:55.545988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:55.546449 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:55.546450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:55.546498 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:55.546503 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 22:45:55.546651 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 22:45:55.546652 (+ 1us) client_negotiation.cc:769] Sending connection context
0807 22:45:55.546684 (+ 32us) client_negotiation.cc:240] Negotiation successful
0807 22:45:55.546722 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260807 22:45:55.556638 24518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:55.556775 24518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:45:55.557476 24518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08c5e1345a704743aa11e1f790451afd: Sending DeleteTablet for 3 replicas of tablet 7737780ed1da4ccc8802a47411403301
I20260807 22:45:55.557802 24801 tablet_service.cc:1514] Processing DeleteTablet for tablet 7737780ed1da4ccc8802a47411403301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:55 UTC) from {username='oryx', principal='oryx/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:59656
I20260807 22:45:55.557812 24945 tablet_service.cc:1514] Processing DeleteTablet for tablet 7737780ed1da4ccc8802a47411403301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:45:55 UTC) from {username='oryx', principal='oryx/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:54794
I20260807 22:45:55.557912 25067 tablet_replica.cc:330] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599: stopping tablet replica
I20260807 22:45:55.557915 25068 tablet_replica.cc:330] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708: stopping tablet replica
I20260807 22:45:55.557963 25067 raft_consensus.cc:2229] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:45:55.557976 25068 raft_consensus.cc:2229] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:45:55.558012 25068 raft_consensus.cc:2258] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:55.558053 25067 raft_consensus.cc:2258] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:45:55.558399 25067 ts_tablet_manager.cc:1905] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:55.558358 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 24593
I20260807 22:45:55.558486 25068 ts_tablet_manager.cc:1905] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:45:55.560135 25067 ts_tablet_manager.cc:1918] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:45:55.560236 25067 log.cc:1196] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/7737780ed1da4ccc8802a47411403301
I20260807 22:45:55.560398 25068 ts_tablet_manager.cc:1918] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 22:45:55.560456 25068 log.cc:1196] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/7737780ed1da4ccc8802a47411403301
I20260807 22:45:55.560499 25067 ts_tablet_manager.cc:1939] T 7737780ed1da4ccc8802a47411403301 P 68aa9dace3cd4d479522e301c9245599: Deleting consensus metadata
I20260807 22:45:55.560698 25068 ts_tablet_manager.cc:1939] T 7737780ed1da4ccc8802a47411403301 P a8054e282ef845688c49c3d68b0fb708: Deleting consensus metadata
I20260807 22:45:55.560966 24505 catalog_manager.cc:4841] TS 68aa9dace3cd4d479522e301c9245599 (127.16.98.130:34781): tablet 7737780ed1da4ccc8802a47411403301 (table test-table [id=4a14520c8ebe4adcbb0775b18326eb72]) successfully deleted
I20260807 22:45:55.562688 24506 catalog_manager.cc:4841] TS a8054e282ef845688c49c3d68b0fb708 (127.16.98.131:44647): tablet 7737780ed1da4ccc8802a47411403301 (table test-table [id=4a14520c8ebe4adcbb0775b18326eb72]) successfully deleted
I20260807 22:45:55.569083 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 24733
W20260807 22:45:55.570055 24505 connection.cc:535] client connection to 127.16.98.129:41199 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 22:45:55.570222 24505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 22:45:55.571092 25131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:55.570790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:41199 (local address 127.0.0.1:48158)
0807 22:45:55.570926 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:55.570999 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
W20260807 22:45:55.572088 24505 catalog_manager.cc:4584] TS a6034f1118ee42ccb46aee7eeb4cdde1 (127.16.98.129:41199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7737780ed1da4ccc8802a47411403301: Network error: Client connection negotiation failed: client connection to 127.16.98.129:41199: connect: Connection refused (error 111)
I20260807 22:45:55.579902 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 24878
I20260807 22:45:55.588217 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 24488
2026-08-07T22:45:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2856 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:45:55 dist-test-slave-rgb1 krb5kdc[25134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:45:55 dist-test-slave-rgb1 krb5kdc[25134](info): set up 2 sockets
Aug 07 22:45:55 dist-test-slave-rgb1 krb5kdc[25134](info): commencing operation
krb5kdc: starting...
W20260807 22:45:56.771816 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:45:56 dist-test-slave-rgb1 krb5kdc[25134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:45:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:45:56.918329 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:40457
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:40457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:57.003533 25148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:57.003957 25148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:57.004033 25148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:57.005555 25148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:57.005657 25148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:57.005690 25148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:57.005720 25148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:57.005750 25148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:57.007385 25148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:40457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:40457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:57.007709 25148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:57.007970 25148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:57.013330 25154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:57.013341 25156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:57.013366 25153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:57.013680 25148 server_base.cc:1034] running on GCE node
I20260807 22:45:57.013919 25148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:57.014405 25148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:57.015583 25148 hybrid_clock.cc:648] HybridClock initialized: now 1786142757015572 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-07T22:45:57Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1437 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:45:57 dist-test-slave-rgb1 krb5kdc[25161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:45:57 dist-test-slave-rgb1 krb5kdc[25161](info): set up 2 sockets
Aug 07 22:45:57 dist-test-slave-rgb1 krb5kdc[25161](info): commencing operation
krb5kdc: starting...
W20260807 22:45:58.152539 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:45:58 dist-test-slave-rgb1 krb5kdc[25161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:45:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:45:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:45:58.297338 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:36451
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:58.387533 25176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:58.387809 25176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:58.387846 25176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:58.389109 25176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:58.389187 25176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:58.389213 25176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:58.389236 25176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:58.389266 25176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:58.390807 25176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:36451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:36451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:58.391129 25176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:58.391381 25176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:58.396668 25184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:58.396713 25182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:58.396672 25181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:58.396994 25176 server_base.cc:1034] running on GCE node
I20260807 22:45:58.397296 25176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:58.397641 25176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:58.398864 25176 hybrid_clock.cc:648] HybridClock initialized: now 1786142758398771 us; error 71 us; skew 500 ppm
Aug 07 22:45:58 dist-test-slave-rgb1 krb5kdc[25161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142758, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:58.408092 25176 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:45:58.408922 25176 webserver.cc:460] Webserver started at http://127.16.98.190:43437/ using document root <none> and password file <none>
I20260807 22:45:58.409284 25176 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:58.409365 25176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:58.409529 25176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:58.410682 25176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "af3edd464fb34d4e8f536ba5696163f5"
format_stamp: "Formatted at 2026-08-07 22:45:58 on dist-test-slave-rgb1"
server_key: "fe7ab5c7c42bea904600e5bfed4b5bc7"
server_key_iv: "2b2f45df462e997425fbd92aee216b9f"
server_key_version: "encryptionkey@0"
I20260807 22:45:58.411077 25176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "af3edd464fb34d4e8f536ba5696163f5"
format_stamp: "Formatted at 2026-08-07 22:45:58 on dist-test-slave-rgb1"
server_key: "fe7ab5c7c42bea904600e5bfed4b5bc7"
server_key_iv: "2b2f45df462e997425fbd92aee216b9f"
server_key_version: "encryptionkey@0"
I20260807 22:45:58.413065 25176 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 22:45:58.414304 25191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:58.414745 25176 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:58.414973 25176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "af3edd464fb34d4e8f536ba5696163f5"
format_stamp: "Formatted at 2026-08-07 22:45:58 on dist-test-slave-rgb1"
server_key: "fe7ab5c7c42bea904600e5bfed4b5bc7"
server_key_iv: "2b2f45df462e997425fbd92aee216b9f"
server_key_version: "encryptionkey@0"
I20260807 22:45:58.415117 25176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:58.455292 25176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:58.458365 25176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:58.458583 25176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:58.463451 25176 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:36451
I20260807 22:45:58.463505 25243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:36451 every 8 connection(s)
I20260807 22:45:58.463940 25176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:58.464772 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 25176
I20260807 22:45:58.464977 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:45:58.465201 16778 external_mini_cluster.cc:1428] Setting key d4509fedee01c0ba6c2acf95c76171ed
I20260807 22:45:58.465207 25244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:45:58.468346 25244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5: Bootstrap starting.
I20260807 22:45:58.469193 25244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5: Neither blocks nor log segments found. Creating new log.
I20260807 22:45:58.469539 25244 log.cc:824] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:58.470304 25244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5: No bootstrap required, opened a new log
I20260807 22:45:58.472368 25244 raft_consensus.cc:348] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3edd464fb34d4e8f536ba5696163f5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36451 } }
I20260807 22:45:58.472615 25244 raft_consensus.cc:374] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:45:58.472699 25244 raft_consensus.cc:729] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3edd464fb34d4e8f536ba5696163f5, State: Initialized, Role: FOLLOWER
I20260807 22:45:58.472886 25244 consensus_queue.cc:260] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3edd464fb34d4e8f536ba5696163f5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36451 } }
I20260807 22:45:58.472995 25244 raft_consensus.cc:388] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:58.473043 25244 raft_consensus.cc:482] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:58.473093 25244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 0 FOLLOWER]: Advancing to term 1
Aug 07 22:45:58 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:58.474036 25244 raft_consensus.cc:504] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3edd464fb34d4e8f536ba5696163f5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36451 } }
I20260807 22:45:58.474264 25244 leader_election.cc:302] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af3edd464fb34d4e8f536ba5696163f5; no voters:
I20260807 22:45:58.474720 25244 leader_election.cc:288] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:45:58.474769 25249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:45:58.474947 25249 raft_consensus.cc:686] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 1 LEADER]: Becoming Leader. State: Replica: af3edd464fb34d4e8f536ba5696163f5, State: Running, Role: LEADER
I20260807 22:45:58.475128 25244 sys_catalog.cc:564] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:58.475188 25249 consensus_queue.cc:237] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3edd464fb34d4e8f536ba5696163f5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36451 } }
I20260807 22:45:58.476020 25250 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af3edd464fb34d4e8f536ba5696163f5. Latest consensus state: current_term: 1 leader_uuid: "af3edd464fb34d4e8f536ba5696163f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3edd464fb34d4e8f536ba5696163f5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36451 } } }
I20260807 22:45:58.476248 25250 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:58.477089 25251 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af3edd464fb34d4e8f536ba5696163f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3edd464fb34d4e8f536ba5696163f5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36451 } } }
I20260807 22:45:58.477252 25251 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:58.477427 25259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:58.478192 25259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:58.478533 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:58.465942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.16.98.190:36451)
0807 22:45:58.466177 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:58.466191 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:58.466238 (+ 47us) server_negotiation.cc:407] Connection header received
0807 22:45:58.466808 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:58.466815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:58.466866 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:58.467044 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:58.467761 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:58.468876 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:58.469844 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:58.470566 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:58.470920 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:58.474501 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:58.474505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:58.474506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:58.474518 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:58.476500 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:58.477016 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:58.477020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:58.477020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:58.477165 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:58.477414 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:58.477415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:58.477415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:58.477800 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:58.477978 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:58.478136 (+ 158us) server_negotiation.cc:299] Negotiation successful
0807 22:45:58.478306 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":88,"threads_started":1}
I20260807 22:45:58.479998 25259 catalog_manager.cc:1347] Generated new cluster ID: d19d919113d840dd9431f017b67f19db
I20260807 22:45:58.480072 25259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:45:58.490600 25259 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:45:58.491196 25259 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:58.499526 25259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5: Generated new TSK 0
I20260807 22:45:58.499751 25259 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:45:58.537580 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36451
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:58.638756 25270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:58.639003 25270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:58.639034 25270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:58.640391 25270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:58.640447 25270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:58.640478 25270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:45:58.642282 25270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:58.642582 25270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:58.642868 25270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:58.648686 25279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:58.648852 25270 server_base.cc:1034] running on GCE node
W20260807 22:45:58.648689 25277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:58.648689 25276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:58.649230 25270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:58.649556 25270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:58.650753 25270 hybrid_clock.cc:648] HybridClock initialized: now 1786142758650740 us; error 49 us; skew 500 ppm
Aug 07 22:45:58 dist-test-slave-rgb1 krb5kdc[25161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142758, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:58.653663 25270 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:45:58.654214 25270 webserver.cc:460] Webserver started at http://127.16.98.129:41483/ using document root <none> and password file <none>
I20260807 22:45:58.654477 25270 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:58.654549 25270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:58.654704 25270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:58.655812 25270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "79ff3874673545aea20c244081dd37ad"
format_stamp: "Formatted at 2026-08-07 22:45:58 on dist-test-slave-rgb1"
server_key: "eb638c8b901b55b5743e250c2cdf4e21"
server_key_iv: "397c7d5fcf093185cf72bd001f66fb79"
server_key_version: "encryptionkey@0"
I20260807 22:45:58.656200 25270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "79ff3874673545aea20c244081dd37ad"
format_stamp: "Formatted at 2026-08-07 22:45:58 on dist-test-slave-rgb1"
server_key: "eb638c8b901b55b5743e250c2cdf4e21"
server_key_iv: "397c7d5fcf093185cf72bd001f66fb79"
server_key_version: "encryptionkey@0"
I20260807 22:45:58.657805 25270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:45:58.658784 25286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:58.659055 25270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:58.659178 25270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "79ff3874673545aea20c244081dd37ad"
format_stamp: "Formatted at 2026-08-07 22:45:58 on dist-test-slave-rgb1"
server_key: "eb638c8b901b55b5743e250c2cdf4e21"
server_key_iv: "397c7d5fcf093185cf72bd001f66fb79"
server_key_version: "encryptionkey@0"
I20260807 22:45:58.659265 25270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:58.681830 25270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:58.684695 25270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:58.684898 25270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:58.685194 25270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:58.685559 25270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:58.685621 25270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:58.685667 25270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:58.685696 25270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:58.693243 25270 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:34681
I20260807 22:45:58.693301 25400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:34681 every 8 connection(s)
I20260807 22:45:58.693683 25270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:45:58.694537 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 25270
I20260807 22:45:58.694675 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:45:58.694945 16778 external_mini_cluster.cc:1428] Setting key c149a6a1ba317f9f5e140f2606f5640b
Aug 07 22:45:58 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142758, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
I20260807 22:45:58.705785 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:58.694667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:39347 (local address 127.16.98.190:36451)
0807 22:45:58.694749 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:58.694753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:58.694915 (+ 162us) server_negotiation.cc:407] Connection header received
0807 22:45:58.695777 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:58.695778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:58.695789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:58.695825 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:58.696915 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:58.697482 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:58.698536 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:58.698757 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:58.698836 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:58.701575 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:58.701577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:58.701577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:58.701589 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:58.703572 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:58.704226 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:58.704228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:58.704229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:58.704305 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:58.704689 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:58.704690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:58.704691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:58.704866 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:58.704956 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:58.705533 (+ 577us) server_negotiation.cc:299] Negotiation successful
0807 22:45:58.705675 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 22:45:58.705976 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:58.694575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:39347)
0807 22:45:58.694772 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:58.694818 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:58.695691 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:58.695907 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:58.695911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:58.696534 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:58.696815 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:58.696820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:58.697575 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:58.697576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:58.698413 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:58.698416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:58.698858 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:58.698859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:58.699000 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:58.699583 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:58.699592 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:58.701451 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:58.703738 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:58.703756 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:58.703759 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:58.704121 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:58.704428 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:58.704429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:58.704429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:58.704612 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:58.704972 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:58.704979 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:58.705129 (+ 150us) client_negotiation.cc:769] Sending connection context
0807 22:45:58.705487 (+ 358us) client_negotiation.cc:240] Negotiation successful
0807 22:45:58.705697 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20260807 22:45:58.706883 25401 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36451
I20260807 22:45:58.707016 25401 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:58.707237 25401 heartbeater.cc:511] Master 127.16.98.190:36451 requested a full tablet report, sending...
I20260807 22:45:58.707940 25208 ts_manager.cc:194] Registered new tserver with Master: 79ff3874673545aea20c244081dd37ad (127.16.98.129:34681)
I20260807 22:45:58.708909 25208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:39347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:45:58.728626 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36451
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:58.818990 25406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:58.819257 25406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:58.819298 25406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:58.820531 25406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:58.820627 25406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:58.820678 25406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:45:58.822762 25406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:58.823210 25406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:58.823498 25406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:58.830192 25415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:58.830243 25413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:58.830201 25412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:58.830298 25406 server_base.cc:1034] running on GCE node
I20260807 22:45:58.830623 25406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:58.830893 25406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:58.832104 25406 hybrid_clock.cc:648] HybridClock initialized: now 1786142758832091 us; error 36 us; skew 500 ppm
Aug 07 22:45:58 dist-test-slave-rgb1 krb5kdc[25161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142758, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:58.835491 25406 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:45:58.836020 25406 webserver.cc:460] Webserver started at http://127.16.98.130:34003/ using document root <none> and password file <none>
I20260807 22:45:58.836304 25406 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:58.836400 25406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:58.836663 25406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:58.837880 25406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "006c41cb087f4069bcacb0719e1e3d9f"
format_stamp: "Formatted at 2026-08-07 22:45:58 on dist-test-slave-rgb1"
server_key: "85651096263cc7fbcf11d4ac0297f2e9"
server_key_iv: "768fd6fd46e34694d741a310568dd203"
server_key_version: "encryptionkey@0"
I20260807 22:45:58.838286 25406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "006c41cb087f4069bcacb0719e1e3d9f"
format_stamp: "Formatted at 2026-08-07 22:45:58 on dist-test-slave-rgb1"
server_key: "85651096263cc7fbcf11d4ac0297f2e9"
server_key_iv: "768fd6fd46e34694d741a310568dd203"
server_key_version: "encryptionkey@0"
I20260807 22:45:58.839861 25406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:45:58.840833 25422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:58.841068 25406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:58.841192 25406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "006c41cb087f4069bcacb0719e1e3d9f"
format_stamp: "Formatted at 2026-08-07 22:45:58 on dist-test-slave-rgb1"
server_key: "85651096263cc7fbcf11d4ac0297f2e9"
server_key_iv: "768fd6fd46e34694d741a310568dd203"
server_key_version: "encryptionkey@0"
I20260807 22:45:58.841281 25406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:58.854645 25406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:58.857470 25406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:58.857673 25406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:58.857954 25406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:58.858333 25406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:58.858400 25406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:58.858448 25406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:58.858510 25406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:58.865765 25406 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:33423
I20260807 22:45:58.865914 25537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:33423 every 8 connection(s)
I20260807 22:45:58.866225 25406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
Aug 07 22:45:58 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142758, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:58.875748 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 25406
I20260807 22:45:58.875892 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:45:58.876194 16778 external_mini_cluster.cc:1428] Setting key af4f3abc0c16edd1e53bfe8628bdd8c3
I20260807 22:45:58.878144 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:58.867107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:34171 (local address 127.16.98.190:36451)
0807 22:45:58.867289 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:58.867292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:58.867596 (+ 304us) server_negotiation.cc:407] Connection header received
0807 22:45:58.868276 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:58.868277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:58.868291 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:58.868351 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:58.869695 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:58.870214 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:58.871310 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:58.871527 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:58.871618 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:58.874348 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:58.874350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:58.874350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:58.874360 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:58.876331 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:58.876884 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:58.876886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:58.876886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:58.876974 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:58.877328 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:58.877329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:58.877329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:58.877528 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:58.877602 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:58.877912 (+ 310us) server_negotiation.cc:299] Negotiation successful
0807 22:45:58.878005 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260807 22:45:58.878226 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:58.867135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:34171)
0807 22:45:58.867445 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:58.867489 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:58.868151 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:58.868447 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:58.868451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:58.869320 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:58.869552 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:58.869558 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:58.870316 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:58.870317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:58.871208 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:58.871211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:58.871631 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:58.871632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:58.871765 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:58.872354 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:58.872363 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:58.874196 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:58.876470 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:58.876484 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:58.876486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:58.876797 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:58.877058 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:58.877059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:58.877060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:58.877243 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:58.877612 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:58.877619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:58.877758 (+ 139us) client_negotiation.cc:769] Sending connection context
0807 22:45:58.877849 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 22:45:58.877991 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":98,"threads_started":1}
I20260807 22:45:58.878806 25538 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36451
I20260807 22:45:58.878937 25538 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:58.879155 25538 heartbeater.cc:511] Master 127.16.98.190:36451 requested a full tablet report, sending...
I20260807 22:45:58.879624 25208 ts_manager.cc:194] Registered new tserver with Master: 006c41cb087f4069bcacb0719e1e3d9f (127.16.98.130:33423)
I20260807 22:45:58.880167 25208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:34171
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:45:58.909837 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36451
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:59.008267 25543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:59.008533 25543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:59.008635 25543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:59.010072 25543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:59.010126 25543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:59.010159 25543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:45:59.011940 25543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:59.012181 25543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:59.012394 25543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:59.018095 25548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:59.018177 25549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:59.018173 25551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:59.018525 25543 server_base.cc:1034] running on GCE node
I20260807 22:45:59.018752 25543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:59.019047 25543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:59.020226 25543 hybrid_clock.cc:648] HybridClock initialized: now 1786142759020205 us; error 59 us; skew 500 ppm
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142759, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:59.023231 25543 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:45:59.023842 25543 webserver.cc:460] Webserver started at http://127.16.98.131:41443/ using document root <none> and password file <none>
I20260807 22:45:59.024183 25543 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:59.024272 25543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:59.024442 25543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:45:59.025753 25543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "f312fae2a6ae49a7bc1e9e6c9cc2cc30"
format_stamp: "Formatted at 2026-08-07 22:45:59 on dist-test-slave-rgb1"
server_key: "a3e1c5bf30563d2640c6ee54c7dcfb5c"
server_key_iv: "6363c14fa14a7d5d5d7a15af752815f4"
server_key_version: "encryptionkey@0"
I20260807 22:45:59.026170 25543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "f312fae2a6ae49a7bc1e9e6c9cc2cc30"
format_stamp: "Formatted at 2026-08-07 22:45:59 on dist-test-slave-rgb1"
server_key: "a3e1c5bf30563d2640c6ee54c7dcfb5c"
server_key_iv: "6363c14fa14a7d5d5d7a15af752815f4"
server_key_version: "encryptionkey@0"
I20260807 22:45:59.027755 25543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:45:59.028743 25558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:59.029307 25543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:59.029561 25543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "f312fae2a6ae49a7bc1e9e6c9cc2cc30"
format_stamp: "Formatted at 2026-08-07 22:45:59 on dist-test-slave-rgb1"
server_key: "a3e1c5bf30563d2640c6ee54c7dcfb5c"
server_key_iv: "6363c14fa14a7d5d5d7a15af752815f4"
server_key_version: "encryptionkey@0"
I20260807 22:45:59.029812 25543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:59.055723 25543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:59.058589 25543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:59.058795 25543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:59.059101 25543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:59.059468 25543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:59.059530 25543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:59.059579 25543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:59.059608 25543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:59.067024 25543 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:42299
I20260807 22:45:59.067068 25676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:42299 every 8 connection(s)
I20260807 22:45:59.067469 25543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142759, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:45:59.077497 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 25543
I20260807 22:45:59.077641 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:45:59.077844 16778 external_mini_cluster.cc:1428] Setting key 89cbef951a7c170c6aecc47eedf6d176
I20260807 22:45:59.079114 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:59.068310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:48123 (local address 127.16.98.190:36451)
0807 22:45:59.068418 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:59.068420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:59.068645 (+ 225us) server_negotiation.cc:407] Connection header received
0807 22:45:59.069272 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:59.069273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:59.069286 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:59.069344 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:59.070381 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.070888 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.071926 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.072129 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.072206 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.074762 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:59.074765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:59.074765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:59.074777 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:59.077258 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:59.078036 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:59.078037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:59.078037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:59.078110 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:59.078391 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:59.078392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:59.078392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:59.078523 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:59.078602 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:59.078929 (+ 327us) server_negotiation.cc:299] Negotiation successful
0807 22:45:59.079021 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 22:45:59.079332 25679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:59.068226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:48123)
0807 22:45:59.068459 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:59.068513 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:59.069154 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:59.069424 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:59.069429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:59.070054 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:59.070269 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.070274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.070991 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.070992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.071823 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.071825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.072227 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.072228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.072360 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.072950 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:59.072959 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:59.074621 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:45:59.077494 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:59.077504 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:59.077506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:59.077954 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:59.078185 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:45:59.078186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:45:59.078187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:45:59.078313 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:45:59.078605 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:45:59.078613 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:45:59.078774 (+ 161us) client_negotiation.cc:769] Sending connection context
0807 22:45:59.078894 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 22:45:59.079076 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
I20260807 22:45:59.080044 25677 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36451
I20260807 22:45:59.080137 25677 heartbeater.cc:464] Registering TS with master...
I20260807 22:45:59.080353 25677 heartbeater.cc:511] Master 127.16.98.190:36451 requested a full tablet report, sending...
I20260807 22:45:59.080878 25208 ts_manager.cc:194] Registered new tserver with Master: f312fae2a6ae49a7bc1e9e6c9cc2cc30 (127.16.98.131:42299)
I20260807 22:45:59.081292 25208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:48123
I20260807 22:45:59.089813 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.129.keytab.
Entry for principal oryx/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.129.keytab.
WARNING: no policy specified for oryx/127.16.98.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.130.keytab.
Entry for principal oryx/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.130.keytab.
WARNING: no policy specified for oryx/127.16.98.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.131.keytab.
Entry for principal oryx/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.131.keytab.
I20260807 22:45:59.188812 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 25270
I20260807 22:45:59.195390 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 25406
I20260807 22:45:59.201833 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 25543
I20260807 22:45:59.207782 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 25176
I20260807 22:45:59.214795 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:36451
--webserver_interface=127.16.98.190
--webserver_port=43437
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:45:59.308348 25686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:59.308735 25686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:59.308781 25686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:59.310223 25686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:45:59.310317 25686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:59.310348 25686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:59.310384 25686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:45:59.310398 25686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:45:59.312249 25686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:36451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:36451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=43437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:59.312525 25686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:59.312808 25686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:59.318176 25694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:59.318243 25692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:59.318370 25691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:59.318625 25686 server_base.cc:1034] running on GCE node
I20260807 22:45:59.318948 25686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:59.319234 25686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:59.320456 25686 hybrid_clock.cc:648] HybridClock initialized: now 1786142759320418 us; error 61 us; skew 500 ppm
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142759, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:59.323660 25686 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:45:59.324219 25686 webserver.cc:460] Webserver started at http://127.16.98.190:43437/ using document root <none> and password file <none>
I20260807 22:45:59.324502 25686 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:59.324633 25686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:59.326412 25686 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:59.327346 25701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:59.327659 25686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:59.327785 25686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "af3edd464fb34d4e8f536ba5696163f5"
format_stamp: "Formatted at 2026-08-07 22:45:58 on dist-test-slave-rgb1"
server_key: "fe7ab5c7c42bea904600e5bfed4b5bc7"
server_key_iv: "2b2f45df462e997425fbd92aee216b9f"
server_key_version: "encryptionkey@0"
I20260807 22:45:59.328258 25686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:59.341782 25686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:59.344798 25686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:59.344991 25686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:59.349798 25686 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:36451
I20260807 22:45:59.349895 25756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:36451 every 8 connection(s)
I20260807 22:45:59.350426 25686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:45:59.350998 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 25686
I20260807 22:45:59.351292 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:34681
--local_ip_for_outbound_sockets=127.16.98.129
--tserver_master_addrs=127.16.98.190:36451
--webserver_port=41483
--webserver_interface=127.16.98.129
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.98.129
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:45:59.352294 25757 sys_catalog.cc:263] Verifying existing consensus state
I20260807 22:45:59.353443 25757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5: Bootstrap starting.
I20260807 22:45:59.365746 25757 log.cc:824] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5: Log is configured to *not* fsync() on all Append() calls
I20260807 22:45:59.368045 25757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:45:59.368395 25757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5: Bootstrap complete.
I20260807 22:45:59.371814 25757 raft_consensus.cc:348] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3edd464fb34d4e8f536ba5696163f5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36451 } }
I20260807 22:45:59.372165 25757 raft_consensus.cc:729] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3edd464fb34d4e8f536ba5696163f5, State: Initialized, Role: FOLLOWER
I20260807 22:45:59.372330 25757 consensus_queue.cc:260] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3edd464fb34d4e8f536ba5696163f5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36451 } }
I20260807 22:45:59.372408 25757 raft_consensus.cc:388] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:45:59.372449 25757 raft_consensus.cc:482] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:45:59.372511 25757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:45:59.374030 25757 raft_consensus.cc:504] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3edd464fb34d4e8f536ba5696163f5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36451 } }
I20260807 22:45:59.374167 25757 leader_election.cc:302] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af3edd464fb34d4e8f536ba5696163f5; no voters:
I20260807 22:45:59.374799 25757 leader_election.cc:288] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 22:45:59.374927 25761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 22:45:59.375120 25761 raft_consensus.cc:686] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [term 2 LEADER]: Becoming Leader. State: Replica: af3edd464fb34d4e8f536ba5696163f5, State: Running, Role: LEADER
I20260807 22:45:59.375313 25761 consensus_queue.cc:237] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3edd464fb34d4e8f536ba5696163f5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36451 } }
I20260807 22:45:59.375130 25757 sys_catalog.cc:564] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:45:59.375813 25762 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "af3edd464fb34d4e8f536ba5696163f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3edd464fb34d4e8f536ba5696163f5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36451 } } }
I20260807 22:45:59.376046 25762 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:59.376421 25767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:45:59.376639 25763 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af3edd464fb34d4e8f536ba5696163f5. Latest consensus state: current_term: 2 leader_uuid: "af3edd464fb34d4e8f536ba5696163f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3edd464fb34d4e8f536ba5696163f5" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36451 } } }
I20260807 22:45:59.376780 25763 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5 [sys.catalog]: This master's current role is: LEADER
I20260807 22:45:59.377229 25767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:45:59.377668 25767 catalog_manager.cc:1259] Loaded cluster ID: d19d919113d840dd9431f017b67f19db
I20260807 22:45:59.377772 25767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:45:59.378706 25767 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:45:59.378914 25767 catalog_manager.cc:5879] T 00000000000000000000000000000000 P af3edd464fb34d4e8f536ba5696163f5: Loaded TSK: 0
I20260807 22:45:59.379163 25767 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 22:45:59.477047 25759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:59.477324 25759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:59.477381 25759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:59.478960 25759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:59.479041 25759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:59.479103 25759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:45:59.480930 25759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.129.keytab
--principal=oryx/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:34681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=41483
--tserver_master_addrs=127.16.98.190:36451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:59.481243 25759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:59.481730 25759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:59.489555 25784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:59.489885 25759 server_base.cc:1034] running on GCE node
W20260807 22:45:59.490072 25786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:59.489547 25783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:59.490453 25759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:59.490808 25759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:59.492089 25759 hybrid_clock.cc:648] HybridClock initialized: now 1786142759492057 us; error 45 us; skew 500 ppm
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142759, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:59.495805 25759 init.cc:377] Logged in from keytab as oryx/127.16.98.129@KRBTEST.COM (short username oryx)
I20260807 22:45:59.496361 25759 webserver.cc:460] Webserver started at http://127.16.98.129:41483/ using document root <none> and password file <none>
I20260807 22:45:59.496676 25759 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:59.496766 25759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:59.498392 25759 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:59.499396 25793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:59.499720 25759 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:59.499836 25759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "79ff3874673545aea20c244081dd37ad"
format_stamp: "Formatted at 2026-08-07 22:45:58 on dist-test-slave-rgb1"
server_key: "eb638c8b901b55b5743e250c2cdf4e21"
server_key_iv: "397c7d5fcf093185cf72bd001f66fb79"
server_key_version: "encryptionkey@0"
I20260807 22:45:59.500308 25759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:59.530613 25759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:59.533587 25759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:59.533788 25759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:59.534126 25759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:59.534516 25759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:59.534578 25759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:59.534627 25759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:59.534658 25759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:59.543216 25759 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:34681
I20260807 22:45:59.543262 25910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:34681 every 8 connection(s)
I20260807 22:45:59.543681 25759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:45:59.549639 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 25759
I20260807 22:45:59.549989 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:33423
--local_ip_for_outbound_sockets=127.16.98.130
--tserver_master_addrs=127.16.98.190:36451
--webserver_port=34003
--webserver_interface=127.16.98.130
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.98.130
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:45:59.554024 25913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:59.544549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:50531)
0807 22:45:59.544771 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:59.544892 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:59.545404 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:59.545904 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:59.545908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:59.546582 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:59.546792 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.546796 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.548695 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.548696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.549602 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.549605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.550290 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.550291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.550439 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.551068 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:59.551077 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:59.553777 (+ 2700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":112,"threads_started":1}
W20260807 22:45:59.554137 25914 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:59.544647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:50531 (local address 127.16.98.190:36451)
0807 22:45:59.544838 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:59.544847 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:59.545042 (+ 195us) server_negotiation.cc:407] Connection header received
0807 22:45:59.545599 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:59.545606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:59.545669 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:59.545821 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:59.546992 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.548396 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.549740 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.550181 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.550314 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.553927 (+ 3613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:50531: BlockingRecv error: recv got EOF from 127.16.98.129:50531 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
W20260807 22:45:59.555111 25911 heartbeater.cc:644] Failed to heartbeat to 127.16.98.190:36451 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.98.190:36451: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:45:59.561404 25913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:59.555677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:49385)
0807 22:45:59.555758 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:59.555773 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:59.555848 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:59.556030 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:59.556032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:59.556219 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:59.556296 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.556299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.557084 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.557085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.558009 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.558012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.558407 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.558408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.558491 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.558971 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:59.558978 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:59.561299 (+ 2321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 22:45:59.561487 25914 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:59.555698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:49385 (local address 127.16.98.190:36451)
0807 22:45:59.555744 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:59.555746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:59.555841 (+ 95us) server_negotiation.cc:407] Connection header received
0807 22:45:59.555919 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:59.555920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:59.555933 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:59.556002 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:59.556383 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.556996 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.558110 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.558323 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.558383 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.561411 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:49385: BlockingRecv error: recv got EOF from 127.16.98.129:49385 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:45:59.567071 25913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:59.561791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:41047)
0807 22:45:59.561927 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:59.561949 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:59.562015 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:59.562165 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:59.562166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:59.562330 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:59.562373 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.562375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.563103 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.563104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.563910 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.563912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.564367 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.564368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.564423 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.565020 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:59.565025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:59.566992 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114}
W20260807 22:45:59.567273 25911 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 22:45:59.567306 25914 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:59.561867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:41047 (local address 127.16.98.190:36451)
0807 22:45:59.561933 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:59.561935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:59.562009 (+ 74us) server_negotiation.cc:407] Connection header received
0807 22:45:59.562079 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:59.562080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:59.562090 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:59.562141 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:59.562507 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.563014 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.564034 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.564286 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.564353 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.567245 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:41047: BlockingRecv error: recv got EOF from 127.16.98.129:41047 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260807 22:45:59.640468 25915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:59.640781 25915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:59.640810 25915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:59.642091 25915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:59.642139 25915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:59.642167 25915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:45:59.643684 25915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.130.keytab
--principal=oryx/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:33423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=34003
--tserver_master_addrs=127.16.98.190:36451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:59.643932 25915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:59.644194 25915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:59.649518 25923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:59.649524 25920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:59.649616 25921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:59.650012 25915 server_base.cc:1034] running on GCE node
I20260807 22:45:59.650219 25915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:59.650430 25915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:59.651592 25915 hybrid_clock.cc:648] HybridClock initialized: now 1786142759651582 us; error 36 us; skew 500 ppm
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142759, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:59.654614 25915 init.cc:377] Logged in from keytab as oryx/127.16.98.130@KRBTEST.COM (short username oryx)
I20260807 22:45:59.655216 25915 webserver.cc:460] Webserver started at http://127.16.98.130:34003/ using document root <none> and password file <none>
I20260807 22:45:59.655464 25915 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:59.655525 25915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:59.657058 25915 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:59.657783 25930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:59.658140 25915 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:45:59.658219 25915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "006c41cb087f4069bcacb0719e1e3d9f"
format_stamp: "Formatted at 2026-08-07 22:45:58 on dist-test-slave-rgb1"
server_key: "85651096263cc7fbcf11d4ac0297f2e9"
server_key_iv: "768fd6fd46e34694d741a310568dd203"
server_key_version: "encryptionkey@0"
I20260807 22:45:59.658607 25915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:59.666428 25915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:59.668919 25915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:59.669049 25915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:59.669265 25915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:59.669539 25915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:59.669572 25915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:59.669596 25915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:59.669608 25915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:59.678296 25915 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:33423
I20260807 22:45:59.678349 26046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:33423 every 8 connection(s)
I20260807 22:45:59.678727 25915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
W20260807 22:45:59.686740 25914 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:59.679575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:60667 (local address 127.16.98.190:36451)
0807 22:45:59.679660 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:59.679663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:59.679888 (+ 225us) server_negotiation.cc:407] Connection header received
0807 22:45:59.680354 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:59.680355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:59.680370 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:59.680435 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:59.681455 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.681997 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.683057 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.683249 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.683322 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.686672 (+ 3350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:60667: BlockingRecv error: recv got EOF from 127.16.98.130:60667 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 22:45:59.686753 26049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:59.679526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:60667)
0807 22:45:59.679733 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:59.679772 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:59.680258 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:59.680486 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:59.680489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:59.681149 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:59.681354 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.681358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.682141 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.682142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.682937 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.682939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.683343 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.683344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.683451 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.684082 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:59.684091 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:59.686500 (+ 2409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":114,"threads_started":1}
W20260807 22:45:59.687115 26047 heartbeater.cc:644] Failed to heartbeat to 127.16.98.190:36451 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.98.190:36451: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
I20260807 22:45:59.687490 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 25915
I20260807 22:45:59.687904 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:42299
--local_ip_for_outbound_sockets=127.16.98.131
--tserver_master_addrs=127.16.98.190:36451
--webserver_port=41443
--webserver_interface=127.16.98.131
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.98.131
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:45:59.693426 26049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:59.687395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:51083)
0807 22:45:59.687430 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:59.687448 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:59.687513 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:59.687689 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:59.687690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:59.687888 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:59.687957 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.687959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.688776 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.688777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.689726 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.689730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.690346 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.690347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.690407 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.691031 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:59.691036 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:59.693329 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260807 22:45:59.693547 25914 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:59.687460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:51083 (local address 127.16.98.190:36451)
0807 22:45:59.687527 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:59.687530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:59.687538 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:45:59.687575 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:59.687576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:59.687586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:59.687658 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:59.688116 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.688694 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.689912 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.690238 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.690325 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.693462 (+ 3137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:51083: BlockingRecv error: recv got EOF from 127.16.98.130:51083 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:45:59.699361 26049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:59.693921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:42023)
0807 22:45:59.694079 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:59.694088 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:59.694136 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:59.694296 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:59.694297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:59.694432 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:59.694480 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.694482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.695469 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.695470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.696051 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.696053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.696623 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.696625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.696693 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.697234 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:59.697240 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:59.699306 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132}
W20260807 22:45:59.699517 25914 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:59.694022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:42023 (local address 127.16.98.190:36451)
0807 22:45:59.694085 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:59.694088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:59.694125 (+ 37us) server_negotiation.cc:407] Connection header received
0807 22:45:59.694182 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:59.694183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:59.694195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:59.694263 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:59.694573 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.695373 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.696139 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.696469 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.696551 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.699429 (+ 2878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:42023: BlockingRecv error: recv got EOF from 127.16.98.130:42023 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260807 22:45:59.699595 26047 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 22:45:59.776937 26050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:45:59.777165 26050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:45:59.777190 26050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:45:59.778419 26050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:45:59.778469 26050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:45:59.778497 26050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:45:59.780054 26050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/krb5kdc/oryx_127.16.98.131.keytab
--principal=oryx/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:42299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=41443
--tserver_master_addrs=127.16.98.190:36451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:45:59.780318 26050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:45:59.780526 26050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:45:59.786001 26055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:59.786008 26056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:45:59.786001 26058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:45:59.786335 26050 server_base.cc:1034] running on GCE node
I20260807 22:45:59.786599 26050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:45:59.786808 26050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:45:59.788017 26050 hybrid_clock.cc:648] HybridClock initialized: now 1786142759787993 us; error 39 us; skew 500 ppm
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142759, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:45:59.791119 26050 init.cc:377] Logged in from keytab as oryx/127.16.98.131@KRBTEST.COM (short username oryx)
I20260807 22:45:59.791642 26050 webserver.cc:460] Webserver started at http://127.16.98.131:41443/ using document root <none> and password file <none>
I20260807 22:45:59.791860 26050 fs_manager.cc:359] Metadata directory not provided
I20260807 22:45:59.791930 26050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:45:59.793540 26050 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:59.794353 26065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:59.794683 26050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:45:59.794809 26050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "f312fae2a6ae49a7bc1e9e6c9cc2cc30"
format_stamp: "Formatted at 2026-08-07 22:45:59 on dist-test-slave-rgb1"
server_key: "a3e1c5bf30563d2640c6ee54c7dcfb5c"
server_key_iv: "6363c14fa14a7d5d5d7a15af752815f4"
server_key_version: "encryptionkey@0"
I20260807 22:45:59.795177 26050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:45:59.810807 26050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:45:59.813179 26050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:45:59.813351 26050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:45:59.813635 26050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:45:59.813966 26050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:45:59.814020 26050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:59.814062 26050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:45:59.814090 26050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:45:59.820297 26050 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:42299
I20260807 22:45:59.820348 26180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:42299 every 8 connection(s)
I20260807 22:45:59.820746 26050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:45:59.825196 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 26050
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
W20260807 22:45:59.829700 25914 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:59.821555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:42311 (local address 127.16.98.190:36451)
0807 22:45:59.821631 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:59.821633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:59.821883 (+ 250us) server_negotiation.cc:407] Connection header received
0807 22:45:59.822444 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:59.822445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:59.822458 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:59.822519 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:59.823478 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.823995 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.825004 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.825216 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.825306 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.829613 (+ 4307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:42311: BlockingRecv error: recv got EOF from 127.16.98.131:42311 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 22:45:59.829707 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:59.821568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:42311)
0807 22:45:59.821761 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:59.821796 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:59.822326 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:59.822594 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:59.822597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:59.823169 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:59.823368 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.823373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.824097 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.824098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.824904 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.824906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.825321 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.825322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.825469 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.826305 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:59.826318 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:59.829498 (+ 3180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":108,"threads_started":1}
W20260807 22:45:59.830041 26181 heartbeater.cc:644] Failed to heartbeat to 127.16.98.190:36451 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.98.190:36451: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:45:59.836040 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:59.830429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:49789)
0807 22:45:59.830562 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:59.830572 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:59.830622 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:59.830753 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:59.830753 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:59.830886 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:59.830931 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.830933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.831727 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.831728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.832656 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.832659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.833174 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.833175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.833259 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.833790 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:59.833795 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:59.835956 (+ 2161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116}
I20260807 22:45:59.836405 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:59.825876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.16.98.190:36451)
0807 22:45:59.826002 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:59.826004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:59.826015 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:45:59.826040 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:59.826041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:59.826054 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:59.826144 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:59.827281 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.828117 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.829335 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.829616 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.829687 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.831352 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:45:59.831357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:45:59.831357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:45:59.831368 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:45:59.833550 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:59.834079 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:59.834083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:59.834084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:59.834216 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:45:59.834555 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:45:59.834557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:45:59.834557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:45:59.835212 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:45:59.835391 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 22:45:59.835553 (+ 162us) server_negotiation.cc:299] Negotiation successful
0807 22:45:59.836274 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
W20260807 22:45:59.836421 25914 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:59.830514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:49789 (local address 127.16.98.190:36451)
0807 22:45:59.830576 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:59.830578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:59.830618 (+ 40us) server_negotiation.cc:407] Connection header received
0807 22:45:59.830666 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:59.830667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:59.830677 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:59.830743 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:59.831039 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.831618 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.832794 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.833078 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.833151 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.836357 (+ 3206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:49789: BlockingRecv error: recv got EOF from 127.16.98.131:49789 (error 108)
Metrics: {"mutex_wait_us":238,"server-negotiator.queue_time_us":39}
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:45:59 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:45:59.841689 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:45:59.836388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:47939)
0807 22:45:59.836448 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0807 22:45:59.836672 (+ 224us) client_negotiation.cc:174] Beginning negotiation
0807 22:45:59.836739 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:45:59.836920 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:45:59.836921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:45:59.837076 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:45:59.837114 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.837115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.838013 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.838014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.838589 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:45:59.838590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.839024 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.839025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:45:59.839075 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.839574 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:45:59.839579 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:45:59.841634 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 22:45:59.841845 26186 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:45:59.836460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:47939 (local address 127.16.98.190:36451)
0807 22:45:59.836485 (+ 25us) server_negotiation.cc:206] Beginning negotiation
0807 22:45:59.836488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:45:59.836766 (+ 278us) server_negotiation.cc:407] Connection header received
0807 22:45:59.836796 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:45:59.836797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:45:59.836811 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:45:59.836873 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:45:59.837239 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.837925 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.838671 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:45:59.838857 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:45:59.838913 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:45:59.841744 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:47939: BlockingRecv error: recv got EOF from 127.16.98.131:47939 (error 108)
Metrics: {"server-negotiator.queue_time_us":9,"thread_start_us":69,"threads_started":1}
W20260807 22:45:59.841925 26181 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 07 22:46:00 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:00 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:00.573972 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:00.567888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:54193)
0807 22:46:00.568107 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:00.568125 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:00.568219 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:00.568654 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:00.568656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:00.568869 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:00.568931 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:00.568934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:00.569902 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:00.569904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:00.570652 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:00.570654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:00.571062 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:00.571063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:00.571125 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:00.571813 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:00.571818 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:00.573867 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
W20260807 22:46:00.574061 26189 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:00.568073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:54193 (local address 127.16.98.190:36451)
0807 22:46:00.568338 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:00.568341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:00.568361 (+ 20us) server_negotiation.cc:407] Connection header received
0807 22:46:00.568403 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:00.568404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:00.568425 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:00.568545 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:00.569099 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:00.569752 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:00.570761 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:00.570977 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:00.571057 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:00.573984 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:54193: BlockingRecv error: recv got EOF from 127.16.98.129:54193 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":88,"threads_started":1}
Aug 07 22:46:00 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:00 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:00.707227 26190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:00.700142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:45577)
0807 22:46:00.700310 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:00.700320 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:00.700413 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:00.700644 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:00.700645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:00.700835 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:00.700892 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:00.700894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:00.701848 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:00.701849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:00.702536 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:00.702539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:00.703090 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:00.703092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:00.703172 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:00.703906 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:00.703921 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:00.707116 (+ 3195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":105,"threads_started":1}
W20260807 22:46:00.707387 26189 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:00.700380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:45577 (local address 127.16.98.190:36451)
0807 22:46:00.700434 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:00.700436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:00.700450 (+ 14us) server_negotiation.cc:407] Connection header received
0807 22:46:00.700474 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:00.700475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:00.700488 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:00.700566 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:00.701057 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:00.701717 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:00.702658 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:00.702875 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:00.702949 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:00.707293 (+ 4344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:45577: BlockingRecv error: recv got EOF from 127.16.98.130:45577 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 07 22:46:00 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:00 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:00.849037 26191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:00.842559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:56941)
0807 22:46:00.842857 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:00.842875 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:00.842967 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:00.843433 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:00.843434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:00.843648 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:00.843700 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:00.843702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:00.844496 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:00.844497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:00.845146 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:00.845148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:00.845549 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:00.845550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:00.845612 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:00.846331 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:00.846337 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:00.848924 (+ 2587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
W20260807 22:46:00.849099 26189 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:00.842735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:56941 (local address 127.16.98.190:36451)
0807 22:46:00.842926 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:00.842928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:00.842940 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:46:00.843303 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:00.843304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:00.843323 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:00.843419 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:00.843787 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:00.844344 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:00.845242 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:00.845459 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:00.845537 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:00.849047 (+ 3510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:56941: BlockingRecv error: recv got EOF from 127.16.98.131:56941 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
Aug 07 22:46:01 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:01 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:01.580686 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:01.575017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:49981)
0807 22:46:01.575194 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:01.575208 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:01.575280 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:01.575463 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:01.575464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:01.575665 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:01.575718 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:01.575720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:01.576739 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:01.576741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:01.577344 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:01.577346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:01.577892 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:01.577893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:01.577976 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:01.578573 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:01.578579 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:01.580551 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
W20260807 22:46:01.580756 26193 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:01.575148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:49981 (local address 127.16.98.190:36451)
0807 22:46:01.575279 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:01.575281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:01.575292 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:46:01.575348 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:01.575349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:01.575382 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:01.575457 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:01.575814 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:01.576637 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:01.577551 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:01.577791 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:01.577878 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:01.580674 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:49981: BlockingRecv error: recv got EOF from 127.16.98.129:49981 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
Aug 07 22:46:01 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:01 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:01.714233 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:01.708134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:55445)
0807 22:46:01.708321 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:01.708334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:01.708487 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:01.708732 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:01.708734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:01.709019 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:01.709097 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:01.709100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:01.709891 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:01.709892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:01.710514 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:01.710517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:01.711069 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:01.711071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:01.711148 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:01.711960 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:01.711966 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:01.714162 (+ 2196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
W20260807 22:46:01.714424 26193 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:01.708229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:55445 (local address 127.16.98.190:36451)
0807 22:46:01.708310 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:01.708313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:01.708458 (+ 145us) server_negotiation.cc:407] Connection header received
0807 22:46:01.708605 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:01.708606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:01.708623 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:01.708689 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:01.709206 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:01.709797 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:01.710627 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:01.710908 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:01.711003 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:01.714345 (+ 3342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:55445: BlockingRecv error: recv got EOF from 127.16.98.130:55445 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 07 22:46:01 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:01 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:01.855171 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:01.849828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:57753)
0807 22:46:01.850042 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:01.850054 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:01.850115 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:01.850227 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:01.850228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:01.850430 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:01.850480 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:01.850481 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:01.851202 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:01.851203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:01.851772 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:01.851773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:01.852179 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:01.852181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:01.852243 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:01.853063 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:01.853068 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:01.855063 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":73,"threads_started":1}
W20260807 22:46:01.855223 26193 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:01.850002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:57753 (local address 127.16.98.190:36451)
0807 22:46:01.850052 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:01.850058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:01.850099 (+ 41us) server_negotiation.cc:407] Connection header received
0807 22:46:01.850168 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:01.850169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:01.850180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:01.850222 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:01.850581 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:01.851101 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:01.851871 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:01.852065 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:01.852144 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:01.855154 (+ 3010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:57753: BlockingRecv error: recv got EOF from 127.16.98.131:57753 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 07 22:46:02 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:02 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:02.587023 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:02.581389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:36561)
0807 22:46:02.581736 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:02.581758 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:02.581885 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:02.582145 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:02.582148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:02.582343 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:02.582397 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:02.582399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:02.583208 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:02.583209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:02.583818 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:02.583820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:02.584246 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:02.584247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:02.584298 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:02.584802 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:02.584807 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:02.586955 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
W20260807 22:46:02.587229 26197 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:02.581521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:36561 (local address 127.16.98.190:36451)
0807 22:46:02.581671 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:02.581673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:02.581888 (+ 215us) server_negotiation.cc:407] Connection header received
0807 22:46:02.581976 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:02.581977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:02.581993 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:02.582064 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:02.582508 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:02.583111 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:02.583937 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:02.584142 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:02.584210 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:02.587128 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:36561: BlockingRecv error: recv got EOF from 127.16.98.129:36561 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
Aug 07 22:46:02 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:02 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:02.722899 26198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:02.714969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:55447)
0807 22:46:02.715237 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:02.715255 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:02.715446 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:02.715729 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:02.715730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:02.715996 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:02.716069 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:02.716071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:02.717221 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:02.717223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:02.718180 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:02.718184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:02.718770 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:02.718773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:02.718857 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:02.719608 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:02.719614 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:02.722781 (+ 3167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":123,"threads_started":1}
W20260807 22:46:02.723016 26197 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:02.715110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:55447 (local address 127.16.98.190:36451)
0807 22:46:02.715232 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:02.715236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:02.715415 (+ 179us) server_negotiation.cc:407] Connection header received
0807 22:46:02.715540 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:02.715542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:02.715560 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:02.715628 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:02.716191 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:02.717117 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:02.718283 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:02.718611 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:02.718710 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:02.722941 (+ 4231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:55447: BlockingRecv error: recv got EOF from 127.16.98.130:55447 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 07 22:46:02 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:02 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:02.861328 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:02.855736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:60821)
0807 22:46:02.855963 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:02.855976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:02.856157 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:02.856341 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:02.856342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:02.856583 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:02.856634 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:02.856637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:02.857389 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:02.857390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:02.857977 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:02.857978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:02.858430 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:02.858432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:02.858505 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:02.859098 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:02.859103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:02.861219 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":104,"threads_started":1}
W20260807 22:46:02.861379 26197 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:02.855842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:60821 (local address 127.16.98.190:36451)
0807 22:46:02.855949 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:02.855952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:02.856135 (+ 183us) server_negotiation.cc:407] Connection header received
0807 22:46:02.856235 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:02.856236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:02.856252 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:02.856333 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:02.856733 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:02.857283 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:02.858077 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:02.858330 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:02.858428 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:02.861307 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:60821: BlockingRecv error: recv got EOF from 127.16.98.131:60821 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 07 22:46:03 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:03 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:03.597307 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:03.587501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:34217)
0807 22:46:03.587659 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:03.587670 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:03.587721 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:03.587865 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:03.587866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:03.588054 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:03.588109 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:03.588111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:03.588878 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:03.588880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:03.589464 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:03.589466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:03.589822 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:03.589823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:03.589879 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:03.594903 (+ 5024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:03.594908 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:03.597240 (+ 2332us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
W20260807 22:46:03.597543 26201 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:03.587580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:34217 (local address 127.16.98.190:36451)
0807 22:46:03.587735 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:03.587737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:03.587745 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:46:03.587767 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:03.587768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:03.587778 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:03.587845 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:03.588219 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:03.588776 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:03.589557 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:03.589751 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:03.589809 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:03.597472 (+ 7663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:34217: BlockingRecv error: recv got EOF from 127.16.98.129:34217 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
Aug 07 22:46:03 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:03 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:03.729445 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:03.723934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:55807)
0807 22:46:03.724296 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:03.724309 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:03.724432 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:03.724675 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:03.724676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:03.724886 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:03.724941 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:03.724944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:03.725687 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:03.725688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:03.726334 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:03.726337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:03.726740 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:03.726741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:03.726811 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:03.727321 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:03.727326 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:03.729361 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":116,"threads_started":1}
W20260807 22:46:03.729560 26201 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:03.724055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:55807 (local address 127.16.98.190:36451)
0807 22:46:03.724283 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:03.724286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:03.724409 (+ 123us) server_negotiation.cc:407] Connection header received
0807 22:46:03.724521 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:03.724523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:03.724538 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:03.724640 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:03.725032 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:03.725589 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:03.726437 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:03.726655 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:03.726742 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:03.729492 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:55807: BlockingRecv error: recv got EOF from 127.16.98.130:55807 (error 108)
Metrics: {"server-negotiator.queue_time_us":187}
Aug 07 22:46:03 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:03 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:03.867921 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:03.862105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:56855)
0807 22:46:03.862270 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:03.862285 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:03.862406 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:03.862680 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:03.862681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:03.862893 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:03.862952 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:03.862955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:03.863699 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:03.863700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:03.864545 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:03.864549 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:03.865020 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:03.865022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:03.865082 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:03.865752 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:03.865758 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:03.867853 (+ 2095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
W20260807 22:46:03.868080 26201 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:03.862192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:56855 (local address 127.16.98.190:36451)
0807 22:46:03.862326 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:03.862328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:03.862394 (+ 66us) server_negotiation.cc:407] Connection header received
0807 22:46:03.862533 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:03.862534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:03.862550 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:03.862628 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:03.863097 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:03.863594 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:03.864696 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:03.864950 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:03.865013 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:03.867994 (+ 2981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:56855: BlockingRecv error: recv got EOF from 127.16.98.131:56855 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 07 22:46:04 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:04 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:04.603518 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:04.597857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:37109)
0807 22:46:04.598050 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:04.598061 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:04.598173 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:04.598446 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:04.598447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:04.598641 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:04.598696 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:04.598697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:04.599588 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:04.599590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:04.600210 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:04.600212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:04.600691 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:04.600693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:04.600745 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:04.601286 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:04.601291 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:04.603457 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":101,"threads_started":1}
W20260807 22:46:04.603716 26205 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:04.597973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:37109 (local address 127.16.98.190:36451)
0807 22:46:04.598105 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:04.598107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:04.598153 (+ 46us) server_negotiation.cc:407] Connection header received
0807 22:46:04.598248 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:04.598249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:04.598262 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:04.598353 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:04.598830 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:04.599480 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:04.600319 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:04.600596 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:04.600668 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:04.603627 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:37109: BlockingRecv error: recv got EOF from 127.16.98.129:37109 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
Aug 07 22:46:04 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:04 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:04.735306 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:04.730053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:35865)
0807 22:46:04.730305 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:04.730317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:04.730405 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:04.730591 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:04.730592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:04.730780 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:04.730831 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:04.730832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:04.731516 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:04.731517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:04.732115 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:04.732117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:04.732482 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:04.732484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:04.732546 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:04.733150 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:04.733155 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:04.735207 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":65,"threads_started":1}
W20260807 22:46:04.735378 26205 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:04.730311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:35865 (local address 127.16.98.190:36451)
0807 22:46:04.730383 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:04.730385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:04.730397 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:46:04.730488 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:04.730493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:04.730507 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:04.730586 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:04.730934 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:04.731431 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:04.732205 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:04.732405 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:04.732484 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:04.735312 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:35865: BlockingRecv error: recv got EOF from 127.16.98.130:35865 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 07 22:46:04 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:04 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:04.875347 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:04.868863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:60233)
0807 22:46:04.869073 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:04.869094 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:04.869237 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:04.869548 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:04.869550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:04.869768 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:04.869824 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:04.869826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:04.870741 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:04.870744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:04.872018 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:04.872025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:04.872496 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:04.872497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:04.872605 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:04.873171 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:04.873176 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:04.875243 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":130,"threads_started":1}
W20260807 22:46:04.875398 26205 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:04.868874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:60233 (local address 127.16.98.190:36451)
0807 22:46:04.869074 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:04.869076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:04.869314 (+ 238us) server_negotiation.cc:407] Connection header received
0807 22:46:04.869358 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:04.869359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:04.869382 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:04.869453 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:04.869986 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:04.870590 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:04.872164 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:04.872399 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:04.872469 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:04.875329 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:60233: BlockingRecv error: recv got EOF from 127.16.98.131:60233 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Aug 07 22:46:05 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:05 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:05.610733 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:05.604260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:56473)
0807 22:46:05.604433 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:05.604443 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:05.604499 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:05.604692 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:05.604694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:05.604962 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:05.605023 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:05.605026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:05.606105 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:05.606106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:05.606761 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:05.606764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:05.607407 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:05.607408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:05.607486 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:05.608168 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:05.608173 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:05.610668 (+ 2495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
W20260807 22:46:05.610961 26210 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:05.604355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:56473 (local address 127.16.98.190:36451)
0807 22:46:05.604516 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:05.604518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:05.604534 (+ 16us) server_negotiation.cc:407] Connection header received
0807 22:46:05.604585 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:05.604586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:05.604596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:05.604674 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:05.605201 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:05.606003 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:05.606930 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:05.607308 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:05.607382 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:05.610860 (+ 3478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:56473: BlockingRecv error: recv got EOF from 127.16.98.129:56473 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
Aug 07 22:46:05 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:05 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:05.741255 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:05.735890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:43699)
0807 22:46:05.736129 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:05.736142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:05.736246 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:05.736415 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:05.736416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:05.736648 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:05.736700 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:05.736702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:05.737425 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:05.737426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:05.737970 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:05.737971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:05.738345 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:05.738346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:05.738403 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:05.738936 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:05.738941 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:05.741152 (+ 2211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":112,"threads_started":1}
W20260807 22:46:05.741312 26210 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:05.735995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:43699 (local address 127.16.98.190:36451)
0807 22:46:05.736117 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:05.736120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:05.736226 (+ 106us) server_negotiation.cc:407] Connection header received
0807 22:46:05.736309 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:05.736310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:05.736327 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:05.736409 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:05.736788 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:05.737330 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:05.738063 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:05.738255 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:05.738343 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:05.741242 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:43699: BlockingRecv error: recv got EOF from 127.16.98.130:43699 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 07 22:46:05 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:05 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:05.881605 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:05.876001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:53189)
0807 22:46:05.876231 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:05.876244 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:05.876348 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:05.876602 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:05.876603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:05.876802 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:05.876849 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:05.876851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:05.877911 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:05.877912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:05.878467 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:05.878469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:05.878857 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:05.878858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:05.878919 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:05.879399 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:05.879404 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:05.881530 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":130,"threads_started":1}
W20260807 22:46:05.881778 26210 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:05.876154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:53189 (local address 127.16.98.190:36451)
0807 22:46:05.876232 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:05.876234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:05.876326 (+ 92us) server_negotiation.cc:407] Connection header received
0807 22:46:05.876487 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:05.876488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:05.876502 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:05.876601 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:05.877045 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:05.877839 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:05.878559 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:05.878767 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:05.878855 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:05.881671 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:53189: BlockingRecv error: recv got EOF from 127.16.98.131:53189 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 07 22:46:06 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:06 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:06.616880 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:06.611404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:56077)
0807 22:46:06.611624 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:06.611639 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:06.611739 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:06.611911 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:06.611912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:06.612112 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:06.612166 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:06.612168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:06.612966 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:06.612967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:06.613723 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:06.613726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:06.614126 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:06.614127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:06.614169 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:06.614708 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:06.614713 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:06.616818 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":115,"threads_started":1}
W20260807 22:46:06.617064 26215 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:06.611575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:56077 (local address 127.16.98.190:36451)
0807 22:46:06.611756 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:06.611758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:06.611769 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:46:06.611796 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:06.611797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:06.611813 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:06.611886 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:06.612259 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:06.612850 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:06.613804 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:06.614037 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:06.614101 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:06.616967 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:56077: BlockingRecv error: recv got EOF from 127.16.98.129:56077 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
Aug 07 22:46:06 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:06 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:06.747386 26216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:06.741970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:56329)
0807 22:46:06.742203 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:06.742216 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:06.742316 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:06.742506 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:06.742507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:06.742715 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:06.742765 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:06.742767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:06.743501 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:06.743502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:06.744059 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:06.744061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:06.744431 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:06.744432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:06.744490 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:06.745144 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:06.745150 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:06.747315 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":112,"threads_started":1}
W20260807 22:46:06.747516 26215 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:06.742083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:56329 (local address 127.16.98.190:36451)
0807 22:46:06.742180 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:06.742183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:06.742293 (+ 110us) server_negotiation.cc:407] Connection header received
0807 22:46:06.742407 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:06.742408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:06.742428 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:06.742496 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:06.742847 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:06.743397 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:06.744151 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:06.744346 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:06.744426 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:06.747417 (+ 2991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:56329: BlockingRecv error: recv got EOF from 127.16.98.130:56329 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 07 22:46:06 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:06 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:06.887861 26217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:06.882196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:38109)
0807 22:46:06.882418 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:06.882428 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:06.882511 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:06.882681 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:06.882682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:06.882885 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:06.882935 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:06.882937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:06.883646 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:06.883647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:06.884160 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:06.884161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:06.884533 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:06.884534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:06.884643 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:06.885193 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:06.885198 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:06.887778 (+ 2580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":125,"threads_started":1}
W20260807 22:46:06.888013 26215 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:06.882441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:38109 (local address 127.16.98.190:36451)
0807 22:46:06.882517 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:06.882519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:06.882529 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:46:06.882578 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:06.882578 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:06.882594 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:06.882675 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:06.883026 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:06.883549 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:06.884249 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:06.884455 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:06.884535 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:06.887925 (+ 3390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:38109: BlockingRecv error: recv got EOF from 127.16.98.131:38109 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 07 22:46:07 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:07 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:07.622932 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:07.617431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:56037)
0807 22:46:07.617605 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:07.617621 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:07.617704 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:07.617961 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:07.617963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:07.618140 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:07.618192 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:07.618194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:07.618980 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:07.618981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:07.619608 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:07.619609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:07.620042 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:07.620043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:07.620094 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:07.620635 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:07.620640 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:07.622849 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
W20260807 22:46:07.623109 26219 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:07.617600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:56037 (local address 127.16.98.190:36451)
0807 22:46:07.617742 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:07.617743 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:07.617751 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:46:07.617773 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:07.617773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:07.617785 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:07.617848 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:07.618266 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:07.618810 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:07.619703 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:07.619928 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:07.620001 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:07.623001 (+ 3000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:56037: BlockingRecv error: recv got EOF from 127.16.98.129:56037 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
Aug 07 22:46:07 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:07 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:07.753592 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:07.748144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:58867)
0807 22:46:07.748420 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:07.748440 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:07.748541 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:07.748807 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:07.748808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:07.749068 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:07.749120 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:07.749122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:07.749955 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:07.749956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:07.750488 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:07.750489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:07.750866 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:07.750867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:07.750925 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:07.751461 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:07.751466 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:07.753481 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":113,"threads_started":1}
W20260807 22:46:07.753651 26219 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:07.748262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:58867 (local address 127.16.98.190:36451)
0807 22:46:07.748366 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:07.748369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:07.748518 (+ 149us) server_negotiation.cc:407] Connection header received
0807 22:46:07.748677 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:07.748678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:07.748692 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:07.748792 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:07.749215 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:07.749865 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:07.750597 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:07.750790 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:07.750868 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:07.753601 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:58867: BlockingRecv error: recv got EOF from 127.16.98.130:58867 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 07 22:46:07 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:07 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:07.894280 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:07.888738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:58247)
0807 22:46:07.888860 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:07.888870 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:07.888938 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:07.889075 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:07.889075 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:07.889246 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:07.889296 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:07.889298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:07.890324 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:07.890325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:07.890902 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:07.890904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:07.891328 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:07.891329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:07.891373 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:07.891966 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:07.891971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:07.894178 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
W20260807 22:46:07.894330 26219 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:07.888678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:58247 (local address 127.16.98.190:36451)
0807 22:46:07.888752 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:07.888754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:07.888953 (+ 199us) server_negotiation.cc:407] Connection header received
0807 22:46:07.888984 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:07.888985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:07.889000 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:07.889059 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:07.889425 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:07.890217 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:07.890995 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:07.891247 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:07.891314 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:07.894263 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:58247: BlockingRecv error: recv got EOF from 127.16.98.131:58247 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 07 22:46:08 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:08 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:08.632462 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:08.623541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:38275)
0807 22:46:08.623694 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:08.623706 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:08.623775 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:08.623972 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:08.623973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:08.624192 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:08.624272 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:08.624274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:08.625175 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:08.625176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:08.625781 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:08.625783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:08.626159 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:08.626160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:08.626205 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:08.630150 (+ 3945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:08.630157 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:08.632372 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
W20260807 22:46:08.632622 26223 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:08.623635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:38275 (local address 127.16.98.190:36451)
0807 22:46:08.623790 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:08.623791 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:08.623818 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:46:08.623838 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:08.623839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:08.623850 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:08.623905 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:08.624434 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:08.625059 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:08.625879 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:08.626082 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:08.626143 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:08.632507 (+ 6364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:38275: BlockingRecv error: recv got EOF from 127.16.98.129:38275 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
Aug 07 22:46:08 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:08 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:08.760113 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:08.754268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:56093)
0807 22:46:08.754489 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:08.754504 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:08.754590 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:08.754920 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:08.754922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:08.755202 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:08.755277 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:08.755280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:08.756032 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:08.756033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:08.756694 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:08.756697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:08.757081 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:08.757082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:08.757132 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:08.757681 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:08.757686 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:08.760042 (+ 2356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
W20260807 22:46:08.760313 26223 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:08.754405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:56093 (local address 127.16.98.190:36451)
0807 22:46:08.754505 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:08.754508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:08.754580 (+ 72us) server_negotiation.cc:407] Connection header received
0807 22:46:08.754696 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:08.754696 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:08.754711 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:08.754808 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:08.755438 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:08.755940 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:08.756783 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:08.756967 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:08.757063 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:08.760213 (+ 3150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:56093: BlockingRecv error: recv got EOF from 127.16.98.130:56093 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 07 22:46:08 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:08 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:08.900426 26225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:08.894998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:35361)
0807 22:46:08.895176 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:08.895187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:08.895249 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:08.895499 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:08.895500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:08.895728 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:08.895788 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:08.895791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:08.896649 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:08.896650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:08.897266 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:08.897268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:08.897724 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:08.897726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:08.897787 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:08.898382 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:08.898386 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:08.900300 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":107,"threads_started":1}
W20260807 22:46:08.900525 26223 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:08.895107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:35361 (local address 127.16.98.190:36451)
0807 22:46:08.895182 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:08.895184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:08.895247 (+ 63us) server_negotiation.cc:407] Connection header received
0807 22:46:08.895298 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:08.895299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:08.895313 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:08.895377 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:08.895955 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:08.896507 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:08.897372 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:08.897591 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:08.897656 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:08.900430 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:35361: BlockingRecv error: recv got EOF from 127.16.98.131:35361 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 07 22:46:09 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:09 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:09.637952 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:09.633043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:38165)
0807 22:46:09.633251 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:09.633267 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:09.633335 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:09.633491 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:09.633493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:09.633674 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:09.633736 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:09.633737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:09.634413 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:09.634414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:09.634948 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:09.634950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:09.635329 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:09.635330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:09.635376 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:09.636064 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:09.636068 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:09.637880 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
W20260807 22:46:09.638144 26227 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:09.633173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:38165 (local address 127.16.98.190:36451)
0807 22:46:09.633286 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:09.633288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:09.633318 (+ 30us) server_negotiation.cc:407] Connection header received
0807 22:46:09.633379 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:09.633380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:09.633390 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:09.633449 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:09.633816 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:09.634333 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:09.635059 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:09.635262 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:09.635313 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:09.638028 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:38165: BlockingRecv error: recv got EOF from 127.16.98.129:38165 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
Aug 07 22:46:09 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:09 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:09.766408 26228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:09.760774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:53145)
0807 22:46:09.760928 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:09.760944 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:09.761144 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:09.761344 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:09.761345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:09.761593 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:09.761645 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:09.761647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:09.762461 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:09.762463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:09.763066 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:09.763067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:09.763470 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:09.763471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:09.763532 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:09.764202 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:09.764208 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:09.766321 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
W20260807 22:46:09.766609 26227 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:09.760903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:53145 (local address 127.16.98.190:36451)
0807 22:46:09.760974 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:09.760976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:09.761118 (+ 142us) server_negotiation.cc:407] Connection header received
0807 22:46:09.761234 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:09.761235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:09.761257 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:09.761337 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:09.761767 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:09.762364 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:09.763167 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:09.763383 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:09.763472 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:09.766523 (+ 3051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:53145: BlockingRecv error: recv got EOF from 127.16.98.130:53145 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 07 22:46:09 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:09 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:09.906742 26229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:09.901172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:60939)
0807 22:46:09.901340 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:09.901359 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:09.901428 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:09.901585 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:09.901585 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:09.901758 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:09.901819 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:09.901820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:09.902589 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:09.902590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:09.903257 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:09.903260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:09.903654 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:09.903655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:09.903700 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:09.904594 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:09.904599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:09.906675 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
W20260807 22:46:09.906985 26227 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:09.901304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:60939 (local address 127.16.98.190:36451)
0807 22:46:09.901375 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:09.901377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:09.901404 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:46:09.901494 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:09.901495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:09.901507 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:09.901577 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:09.901909 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:09.902496 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:09.903360 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:09.903573 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:09.903653 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:09.906795 (+ 3142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:60939: BlockingRecv error: recv got EOF from 127.16.98.131:60939 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 07 22:46:10 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:10 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:10.644524 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:10.638530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:35533)
0807 22:46:10.638682 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:10.638693 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:10.638749 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:10.639314 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:10.639316 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:10.639523 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:10.639589 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:10.639592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:10.640465 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:10.640466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:10.641220 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:10.641223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:10.641687 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:10.641688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:10.641738 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:10.642465 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:10.642474 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:10.644444 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
W20260807 22:46:10.644788 26231 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:10.638865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:35533 (local address 127.16.98.190:36451)
0807 22:46:10.639067 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:10.639069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:10.639085 (+ 16us) server_negotiation.cc:407] Connection header received
0807 22:46:10.639120 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:10.639121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:10.639136 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:10.639254 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:10.639811 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:10.640354 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:10.641348 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:10.641590 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:10.641663 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:10.644704 (+ 3041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:35533: BlockingRecv error: recv got EOF from 127.16.98.129:35533 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":108,"threads_started":1}
Aug 07 22:46:10 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:10 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:10.772535 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:10.766959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:60599)
0807 22:46:10.767137 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:10.767147 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:10.767198 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:10.767358 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:10.767359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:10.767565 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:10.767612 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:10.767614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:10.768494 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:10.768495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:10.769302 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:10.769305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:10.769716 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:10.769717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:10.769766 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:10.770483 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:10.770488 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:10.772449 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":113,"threads_started":1}
W20260807 22:46:10.773094 26231 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:10.767142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:60599 (local address 127.16.98.190:36451)
0807 22:46:10.767194 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:10.767196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:10.767207 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:46:10.767274 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:10.767275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:10.767287 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:10.767344 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:10.767748 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:10.768356 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:10.769437 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:10.769632 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:10.769700 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:10.772810 (+ 3110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:60599: BlockingRecv error: recv got EOF from 127.16.98.130:60599 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 07 22:46:10 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:10 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:10.913317 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:10.907630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:36683)
0807 22:46:10.907885 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:10.907899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:10.907976 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:10.908242 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:10.908243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:10.908518 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:10.908617 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:10.908619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:10.909337 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:10.909338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:10.909982 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:10.909984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:10.910420 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:10.910421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:10.910470 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:10.910962 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:10.910967 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:10.913225 (+ 2258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":116,"threads_started":1}
W20260807 22:46:10.913481 26231 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:10.907733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:36683 (local address 127.16.98.190:36451)
0807 22:46:10.907813 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:10.907815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:10.908083 (+ 268us) server_negotiation.cc:407] Connection header received
0807 22:46:10.908115 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:10.908116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:10.908136 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:10.908205 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:10.908718 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:10.909231 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:10.910124 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:10.910335 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:10.910409 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:10.913378 (+ 2969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:36683: BlockingRecv error: recv got EOF from 127.16.98.131:36683 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 07 22:46:11 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:11 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:11.650128 26234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:11.645151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:48749)
0807 22:46:11.645319 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:11.645330 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:11.645390 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:11.645561 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:11.645562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:11.645742 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:11.645793 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:11.645794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:11.646586 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:11.646587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:11.647182 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:11.647185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:11.647634 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:11.647634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:11.647674 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:11.648159 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:11.648164 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:11.650055 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
W20260807 22:46:11.650322 26235 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:11.645246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:48749 (local address 127.16.98.190:36451)
0807 22:46:11.645422 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:11.645429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:11.645438 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:46:11.645461 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:11.645462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:11.645473 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:11.645544 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:11.645965 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:11.646497 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:11.647279 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:11.647549 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:11.647625 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:11.650239 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:48749: BlockingRecv error: recv got EOF from 127.16.98.129:48749 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
Aug 07 22:46:11 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:11 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:11.779899 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:11.773280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:37853)
0807 22:46:11.773647 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:11.773660 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:11.773733 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:11.774214 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:11.774215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:11.774421 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:11.774472 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:11.774474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:11.775621 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:11.775622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:11.776219 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:11.776222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:11.776941 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:11.776942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:11.776991 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:11.777576 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:11.777581 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:11.779818 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":230,"threads_started":1}
W20260807 22:46:11.780088 26235 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:11.773399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:37853 (local address 127.16.98.190:36451)
0807 22:46:11.773523 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:11.773526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:11.773792 (+ 266us) server_negotiation.cc:407] Connection header received
0807 22:46:11.773859 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:11.773861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:11.773884 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:11.774191 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:11.774604 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:11.775452 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:11.776370 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:11.776846 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:11.776917 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:11.779909 (+ 2992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:37853: BlockingRecv error: recv got EOF from 127.16.98.130:37853 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 07 22:46:11 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:11 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:11.919279 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:11.913980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:43449)
0807 22:46:11.914154 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:11.914166 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:11.914230 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:11.914444 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:11.914445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:11.914678 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:11.914740 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:11.914742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:11.915631 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:11.915632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:11.916159 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:11.916160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:11.916647 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:11.916648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:11.916723 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:11.917356 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:11.917361 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:11.919216 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":103,"threads_started":1}
W20260807 22:46:11.919574 26235 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:11.914145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:43449 (local address 127.16.98.190:36451)
0807 22:46:11.914224 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:11.914226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:11.914237 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:46:11.914302 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:11.914303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:11.914317 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:11.914413 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:11.914876 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:11.915554 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:11.916253 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:11.916488 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:11.916582 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:11.919520 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:43449: BlockingRecv error: recv got EOF from 127.16.98.131:43449 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 07 22:46:12 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:12 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:12.658210 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:12.650683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:37059)
0807 22:46:12.650903 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:12.650916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:12.651003 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:12.651211 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:12.651212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:12.651543 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:12.651613 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:12.651616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:12.652846 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:12.652847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:12.654055 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:12.654059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:12.654569 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:12.654570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:12.654661 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:12.655282 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:12.655288 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:12.658144 (+ 2856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":60,"threads_started":1}
W20260807 22:46:12.658444 26239 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:12.650756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:37059 (local address 127.16.98.190:36451)
0807 22:46:12.650906 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:12.650909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:12.650979 (+ 70us) server_negotiation.cc:407] Connection header received
0807 22:46:12.651080 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:12.651081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:12.651107 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:12.651191 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:12.651713 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:12.652745 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:12.654163 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:12.654473 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:12.654574 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:12.658373 (+ 3799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:37059: BlockingRecv error: recv got EOF from 127.16.98.129:37059 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
Aug 07 22:46:12 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:12 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:12.786789 26240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:12.780415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:40911)
0807 22:46:12.780615 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:12.780627 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:12.780686 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:12.780892 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:12.780893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:12.781092 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:12.781142 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:12.781144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:12.782154 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:12.782156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:12.783054 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:12.783057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:12.783648 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:12.783649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:12.783709 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:12.784281 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:12.784286 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:12.786679 (+ 2393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
W20260807 22:46:12.786969 26239 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:12.780688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:40911 (local address 127.16.98.190:36451)
0807 22:46:12.780740 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:12.780742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:12.780751 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:46:12.780786 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:12.780787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:12.780802 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:12.780865 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:12.781224 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:12.782028 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:12.783147 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:12.783534 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:12.783628 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:12.786820 (+ 3192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:40911: BlockingRecv error: recv got EOF from 127.16.98.130:40911 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 07 22:46:12 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:12 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:12.925382 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:12.920012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:55157)
0807 22:46:12.920165 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:12.920177 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:12.920243 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:12.920434 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:12.920436 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:12.920723 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:12.920774 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:12.920776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:12.921469 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:12.921470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:12.922091 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:12.922093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:12.922504 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:12.922505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:12.922554 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:12.923100 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:12.923105 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:12.925229 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
W20260807 22:46:12.925406 26239 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:12.920130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:55157 (local address 127.16.98.190:36451)
0807 22:46:12.920228 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:12.920230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:12.920245 (+ 15us) server_negotiation.cc:407] Connection header received
0807 22:46:12.920298 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:12.920299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:12.920318 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:12.920385 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:12.920873 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:12.921383 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:12.922206 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:12.922422 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:12.922503 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:12.925327 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:55157: BlockingRecv error: recv got EOF from 127.16.98.131:55157 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 07 22:46:13 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:13 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:13.670410 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:13.658835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:42777)
0807 22:46:13.659027 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:13.659038 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:13.659098 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:13.659270 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:13.659272 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:13.659448 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:13.659498 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:13.659500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:13.660321 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:13.660322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:13.661204 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:13.661208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:13.661692 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:13.661693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:13.661745 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:13.667680 (+ 5935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:13.667687 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:13.670325 (+ 2638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":104,"threads_started":1}
W20260807 22:46:13.670591 26243 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:13.658957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:42777 (local address 127.16.98.190:36451)
0807 22:46:13.659107 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:13.659109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:13.659119 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:46:13.659145 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:13.659146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:13.659158 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:13.659225 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:13.659606 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:13.660210 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:13.661347 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:13.661595 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:13.661684 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:13.670485 (+ 8801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:42777: BlockingRecv error: recv got EOF from 127.16.98.129:42777 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":57,"threads_started":1}
Aug 07 22:46:13 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:13 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:13.792912 26244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:13.787401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:35817)
0807 22:46:13.787607 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:13.787619 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:13.787720 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:13.787904 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:13.787905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:13.788108 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:13.788160 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:13.788162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:13.788856 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:13.788857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:13.789520 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:13.789523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:13.789875 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:13.789876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:13.789925 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:13.790757 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:13.790764 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:13.792839 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":121,"threads_started":1}
W20260807 22:46:13.793136 26243 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:13.787468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:35817 (local address 127.16.98.190:36451)
0807 22:46:13.787516 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:13.787518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:13.787705 (+ 187us) server_negotiation.cc:407] Connection header received
0807 22:46:13.787804 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:13.787805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:13.787821 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:13.787884 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:13.788242 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:13.788774 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:13.789610 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:13.789801 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:13.789860 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:13.793073 (+ 3213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:35817: BlockingRecv error: recv got EOF from 127.16.98.130:35817 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 07 22:46:13 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:13 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.131@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:13.931339 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:13.926017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.131:41229)
0807 22:46:13.926283 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:13.926297 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:13.926404 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:13.926591 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:13.926592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:13.926797 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:13.926915 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:13.926917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:13.927633 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:13.927634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:13.928180 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:13.928182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:13.928617 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:13.928619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:13.928682 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:13.929224 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:13.929229 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:13.931259 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":137,"threads_started":1}
W20260807 22:46:13.931393 26243 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:13.926133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:41229 (local address 127.16.98.190:36451)
0807 22:46:13.926265 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:13.926268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:13.926381 (+ 113us) server_negotiation.cc:407] Connection header received
0807 22:46:13.926472 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:13.926473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:13.926493 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:13.926569 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:13.927017 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:13.927542 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:13.928282 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:13.928478 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:13.928618 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:13.931340 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.131:41229: BlockingRecv error: recv got EOF from 127.16.98.131:41229 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 07 22:46:14 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:14 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.129@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:14.676529 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:14.671109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.129:59893)
0807 22:46:14.671261 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:14.671276 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:14.671326 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:14.671480 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:14.671480 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:14.671651 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:14.671715 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:14.671716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:14.672409 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:14.672410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:14.673054 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:14.673056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:14.673427 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:14.673428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:14.673489 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:14.674116 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:14.674121 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:14.676460 (+ 2339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
W20260807 22:46:14.676745 26247 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:14.671189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:59893 (local address 127.16.98.190:36451)
0807 22:46:14.671347 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:14.671349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:14.671358 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:46:14.671380 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:14.671381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:14.671392 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:14.671466 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:14.671807 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:14.672308 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:14.673139 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:14.673334 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:14.673395 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:14.676641 (+ 3246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.129:59893: BlockingRecv error: recv got EOF from 127.16.98.129:59893 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
Aug 07 22:46:14 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
Aug 07 22:46:14 dist-test-slave-rgb1 krb5kdc[25161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.98.130@KRBTEST.COM for oryx/127.16.98.190@KRBTEST.COM, Server not found in Kerberos database
I20260807 22:46:14.798910 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:14.793588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36451 (local address 127.16.98.130:40719)
0807 22:46:14.793720 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:14.793732 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:14.793803 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:14.793950 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:14.793950 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:14.794176 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:14.794226 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:14.794228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:14.794898 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:14.794899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:14.795489 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:14.795491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:14.795883 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:14.795884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:14.795934 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:14.796628 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:14.796633 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:14.798840 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.98.190:36451: Server oryx/127.16.98.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
W20260807 22:46:14.799162 26247 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:14.793634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:40719 (local address 127.16.98.190:36451)
0807 22:46:14.793743 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:14.793745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:14.793791 (+ 46us) server_negotiation.cc:407] Connection header received
0807 22:46:14.793860 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:14.793860 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:14.793873 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:14.793930 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:14.794301 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:14.794818 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:14.795591 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:14.795783 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:14.795850 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:14.799081 (+ 3231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.98.130:40719: BlockingRecv error: recv got EOF from 127.16.98.130:40719 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 22:46:14.825464 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 25759
I20260807 22:46:14.832803 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 25915
I20260807 22:46:14.838933 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 26050
I20260807 22:46:14.844187 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 25686
2026-08-07T22:46:14Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17819 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-07T22:46:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:46:14Z Disabled control of system clock
I20260807 22:46:14.879141 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:34123
--webserver_interface=127.16.98.190
--webserver_port=0
--builtin_ntp_servers=127.16.98.148:42465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:34123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 22:46:14.964179 26252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:46:14.964419 26252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:46:14.964442 26252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:46:14.965718 26252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:46:14.965770 26252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:46:14.965782 26252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:46:14.965795 26252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:46:14.965808 26252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:46:14.967155 26252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:34123
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:34123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:46:14.967370 26252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:46:14.967636 26252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:46:14.973220 26260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:14.973214 26257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:14.973315 26252 server_base.cc:1034] running on GCE node
W20260807 22:46:14.973318 26258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:14.973760 26252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:46:14.974103 26252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:46:14.975301 26252 hybrid_clock.cc:648] HybridClock initialized: now 1786142774975266 us; error 49 us; skew 500 ppm
I20260807 22:46:14.976868 26252 webserver.cc:460] Webserver started at http://127.16.98.190:35357/ using document root <none> and password file <none>
I20260807 22:46:14.977214 26252 fs_manager.cc:359] Metadata directory not provided
I20260807 22:46:14.977306 26252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:46:14.977497 26252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:46:14.978752 26252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "de1f90c1f87d4732bb998925f12c5711"
format_stamp: "Formatted at 2026-08-07 22:46:14 on dist-test-slave-rgb1"
server_key: "2e40ae7f65c15daa5b1f5c16adb50827"
server_key_iv: "4e923f6655753f31b4f56c651f749c3f"
server_key_version: "encryptionkey@0"
I20260807 22:46:14.979175 26252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "de1f90c1f87d4732bb998925f12c5711"
format_stamp: "Formatted at 2026-08-07 22:46:14 on dist-test-slave-rgb1"
server_key: "2e40ae7f65c15daa5b1f5c16adb50827"
server_key_iv: "4e923f6655753f31b4f56c651f749c3f"
server_key_version: "encryptionkey@0"
I20260807 22:46:14.980873 26252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260807 22:46:14.981992 26266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:14.982522 26252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:46:14.982640 26252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "de1f90c1f87d4732bb998925f12c5711"
format_stamp: "Formatted at 2026-08-07 22:46:14 on dist-test-slave-rgb1"
server_key: "2e40ae7f65c15daa5b1f5c16adb50827"
server_key_iv: "4e923f6655753f31b4f56c651f749c3f"
server_key_version: "encryptionkey@0"
I20260807 22:46:14.982722 26252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:46:14.999851 26252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:46:15.000363 26252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:46:15.000732 26252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:46:15.007623 26252 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:34123
I20260807 22:46:15.007668 26318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:34123 every 8 connection(s)
I20260807 22:46:15.008489 26252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:46:15.009928 26319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:46:15.012914 26319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711: Bootstrap starting.
I20260807 22:46:15.013875 26319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711: Neither blocks nor log segments found. Creating new log.
I20260807 22:46:15.014192 26319 log.cc:824] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711: Log is configured to *not* fsync() on all Append() calls
I20260807 22:46:15.014818 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 26252
I20260807 22:46:15.014947 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:46:15.015031 26319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711: No bootstrap required, opened a new log
I20260807 22:46:15.015355 16778 external_mini_cluster.cc:1428] Setting key 046a84554feb77807135763c879f220d
I20260807 22:46:15.017709 26319 raft_consensus.cc:348] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1f90c1f87d4732bb998925f12c5711" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34123 } }
I20260807 22:46:15.017930 26319 raft_consensus.cc:374] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:46:15.017976 26319 raft_consensus.cc:729] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de1f90c1f87d4732bb998925f12c5711, State: Initialized, Role: FOLLOWER
I20260807 22:46:15.018121 26319 consensus_queue.cc:260] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1f90c1f87d4732bb998925f12c5711" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34123 } }
I20260807 22:46:15.018257 26319 raft_consensus.cc:388] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:46:15.018306 26319 raft_consensus.cc:482] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:46:15.018363 26319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:46:15.019006 26319 raft_consensus.cc:504] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1f90c1f87d4732bb998925f12c5711" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34123 } }
I20260807 22:46:15.019135 26319 leader_election.cc:302] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de1f90c1f87d4732bb998925f12c5711; no voters:
I20260807 22:46:15.019364 26319 leader_election.cc:288] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:46:15.019402 26324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:46:15.019537 26324 raft_consensus.cc:686] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [term 1 LEADER]: Becoming Leader. State: Replica: de1f90c1f87d4732bb998925f12c5711, State: Running, Role: LEADER
I20260807 22:46:15.019644 26324 consensus_queue.cc:237] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1f90c1f87d4732bb998925f12c5711" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34123 } }
I20260807 22:46:15.019699 26319 sys_catalog.cc:564] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:46:15.020021 26325 sys_catalog.cc:455] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de1f90c1f87d4732bb998925f12c5711" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1f90c1f87d4732bb998925f12c5711" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34123 } } }
I20260807 22:46:15.020038 26326 sys_catalog.cc:455] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de1f90c1f87d4732bb998925f12c5711. Latest consensus state: current_term: 1 leader_uuid: "de1f90c1f87d4732bb998925f12c5711" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de1f90c1f87d4732bb998925f12c5711" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34123 } } }
I20260807 22:46:15.020236 26325 sys_catalog.cc:458] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [sys.catalog]: This master's current role is: LEADER
I20260807 22:46:15.020257 26326 sys_catalog.cc:458] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711 [sys.catalog]: This master's current role is: LEADER
W20260807 22:46:15.021817 26340 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:46:15.021884 26340 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:46:15.021950 26333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:46:15.022154 26333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:46:15.023456 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:15.016122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.16.98.190:34123)
0807 22:46:15.016362 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:15.016373 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:15.016410 (+ 37us) server_negotiation.cc:407] Connection header received
0807 22:46:15.017651 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:15.017661 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:15.017756 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:15.017936 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:15.018288 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:15.019862 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:15.021001 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:15.021717 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:15.021845 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:15.022685 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:46:15.022689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:46:15.022689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:46:15.022928 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:46:15.022969 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:46:15.022984 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:46:15.023059 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:15.023164 (+ 105us) server_negotiation.cc:299] Negotiation successful
0807 22:46:15.023250 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20260807 22:46:15.024739 26333 catalog_manager.cc:1347] Generated new cluster ID: 794dac2ec5dc49b5966406118d80d338
I20260807 22:46:15.024835 26333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:46:15.039291 26333 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:46:15.039880 26333 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:46:15.044391 26333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de1f90c1f87d4732bb998925f12c5711: Generated new TSK 0
I20260807 22:46:15.044624 26333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:46:15.052495 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34123
--builtin_ntp_servers=127.16.98.148:42465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260807 22:46:15.145033 26343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:46:15.145280 26343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:46:15.145319 26343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:46:15.146564 26343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:46:15.146611 26343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:46:15.146639 26343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:46:15.148037 26343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:46:15.148319 26343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:46:15.148710 26343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:46:15.154021 26352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:15.154017 26350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:15.154021 26349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:15.154625 26343 server_base.cc:1034] running on GCE node
I20260807 22:46:15.154887 26343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:46:15.155182 26343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:46:15.156370 26343 hybrid_clock.cc:648] HybridClock initialized: now 1786142775156356 us; error 27 us; skew 500 ppm
I20260807 22:46:15.157968 26343 webserver.cc:460] Webserver started at http://127.16.98.129:42975/ using document root <none> and password file <none>
I20260807 22:46:15.158269 26343 fs_manager.cc:359] Metadata directory not provided
I20260807 22:46:15.158344 26343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:46:15.158504 26343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:46:15.159719 26343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "8f1f999b39d14656bcf874cd6d62a577"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "2a97a611ac3abb7cc877bbd545e48292"
server_key_iv: "6183aa61cb61943ad8921ccf4fcd5520"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.160130 26343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "8f1f999b39d14656bcf874cd6d62a577"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "2a97a611ac3abb7cc877bbd545e48292"
server_key_iv: "6183aa61cb61943ad8921ccf4fcd5520"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.161796 26343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260807 22:46:15.162725 26358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.163017 26343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:46:15.163137 26343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "8f1f999b39d14656bcf874cd6d62a577"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "2a97a611ac3abb7cc877bbd545e48292"
server_key_iv: "6183aa61cb61943ad8921ccf4fcd5520"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.163232 26343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:46:15.174010 26343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:46:15.174384 26343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:46:15.174561 26343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:46:15.174822 26343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:46:15.175196 26343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:46:15.175251 26343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.175297 26343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:46:15.175323 26343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.182621 26343 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:44559
I20260807 22:46:15.182672 26476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:44559 every 8 connection(s)
I20260807 22:46:15.183068 26343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:46:15.189092 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 26343
I20260807 22:46:15.189204 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:46:15.189430 16778 external_mini_cluster.cc:1428] Setting key 00bd8c3b86109156e25d91ff6fcea8b8
I20260807 22:46:15.189684 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:15.183970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:55445 (local address 127.16.98.190:34123)
0807 22:46:15.184050 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:15.184052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:15.184304 (+ 252us) server_negotiation.cc:407] Connection header received
0807 22:46:15.184932 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:15.184933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:15.184944 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:15.185020 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:15.185833 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:15.186392 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:15.187714 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:15.187904 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:15.187974 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:15.189152 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:46:15.189153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:46:15.189154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:46:15.189160 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:46:15.189209 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:46:15.189215 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:46:15.189290 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:15.189418 (+ 128us) server_negotiation.cc:299] Negotiation successful
0807 22:46:15.189451 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 22:46:15.189919 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:15.183943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:34123 (local address 127.16.98.129:55445)
0807 22:46:15.184152 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:15.184194 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:15.184826 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:15.185092 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:15.185096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:15.185104 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:15.185644 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:15.185650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:15.186531 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:15.186532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:15.187616 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:15.187619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:15.188026 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:15.188028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:15.188152 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:15.188894 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:46:15.188896 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:15.188938 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:46:15.188977 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:46:15.189306 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:46:15.189320 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:46:15.189324 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:46:15.189420 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 22:46:15.189681 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260807 22:46:15.190490 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34123
--builtin_ntp_servers=127.16.98.148:42465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 22:46:15.190527 26477 heartbeater.cc:347] Connected to a master server at 127.16.98.190:34123
I20260807 22:46:15.190637 26477 heartbeater.cc:464] Registering TS with master...
I20260807 22:46:15.190829 26477 heartbeater.cc:511] Master 127.16.98.190:34123 requested a full tablet report, sending...
I20260807 22:46:15.191573 26283 ts_manager.cc:194] Registered new tserver with Master: 8f1f999b39d14656bcf874cd6d62a577 (127.16.98.129:44559)
I20260807 22:46:15.192479 26283 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.98.129:55445
W20260807 22:46:15.278784 26480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:46:15.279027 26480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:46:15.279072 26480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:46:15.280239 26480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:46:15.280303 26480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:46:15.280345 26480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:46:15.281831 26480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:46:15.282130 26480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:46:15.282456 26480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:46:15.288093 26489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:15.288146 26487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:15.288360 26480 server_base.cc:1034] running on GCE node
W20260807 22:46:15.288463 26486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:15.288774 26480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:46:15.289054 26480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:46:15.290235 26480 hybrid_clock.cc:648] HybridClock initialized: now 1786142775290233 us; error 47 us; skew 500 ppm
I20260807 22:46:15.291785 26480 webserver.cc:460] Webserver started at http://127.16.98.130:39731/ using document root <none> and password file <none>
I20260807 22:46:15.292070 26480 fs_manager.cc:359] Metadata directory not provided
I20260807 22:46:15.292125 26480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:46:15.292271 26480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:46:15.293534 26480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "0c47f57452824b1b9020742947d8066e"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "2a7241b45cf54ae71368cc163dd88d0f"
server_key_iv: "b4474d07265051a8dbd84b586ef86cee"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.293987 26480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "0c47f57452824b1b9020742947d8066e"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "2a7241b45cf54ae71368cc163dd88d0f"
server_key_iv: "b4474d07265051a8dbd84b586ef86cee"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.295576 26480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:46:15.296509 26496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.296736 26480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 22:46:15.296847 26480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "0c47f57452824b1b9020742947d8066e"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "2a7241b45cf54ae71368cc163dd88d0f"
server_key_iv: "b4474d07265051a8dbd84b586ef86cee"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.296931 26480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:46:15.310130 26480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:46:15.310508 26480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:46:15.310681 26480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:46:15.310961 26480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:46:15.311372 26480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:46:15.311434 26480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.311473 26480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:46:15.311506 26480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.317881 26480 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:37187
I20260807 22:46:15.317955 26610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:37187 every 8 connection(s)
I20260807 22:46:15.318339 26480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:46:15.324653 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:15.319334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:34123 (local address 127.16.98.130:59495)
0807 22:46:15.319576 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:15.319614 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:15.320300 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:15.320653 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:15.320657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:15.320666 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:15.321297 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:15.321302 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:15.322107 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:15.322108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:15.322900 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:15.322902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:15.323325 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:15.323327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:15.323452 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:15.324019 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:46:15.324020 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:15.324048 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:46:15.324071 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:46:15.324296 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:46:15.324303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:46:15.324306 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:46:15.324374 (+ 68us) client_negotiation.cc:240] Negotiation successful
0807 22:46:15.324460 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":118,"threads_started":1}
I20260807 22:46:15.324517 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:15.319360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:59495 (local address 127.16.98.190:34123)
0807 22:46:15.319496 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:15.319499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:15.319705 (+ 206us) server_negotiation.cc:407] Connection header received
0807 22:46:15.320440 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:15.320442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:15.320454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:15.320521 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:15.321435 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:15.322012 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:15.323019 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:15.323223 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:15.323289 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:15.324182 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:46:15.324183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:46:15.324184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:46:15.324189 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:46:15.324205 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:46:15.324208 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:46:15.324273 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:15.324421 (+ 148us) server_negotiation.cc:299] Negotiation successful
0807 22:46:15.324447 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 22:46:15.325274 26611 heartbeater.cc:347] Connected to a master server at 127.16.98.190:34123
I20260807 22:46:15.325387 26611 heartbeater.cc:464] Registering TS with master...
I20260807 22:46:15.325630 26611 heartbeater.cc:511] Master 127.16.98.190:34123 requested a full tablet report, sending...
I20260807 22:46:15.326184 26283 ts_manager.cc:194] Registered new tserver with Master: 0c47f57452824b1b9020742947d8066e (127.16.98.130:37187)
I20260807 22:46:15.326659 26283 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.98.130:59495
I20260807 22:46:15.327579 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 26480
I20260807 22:46:15.327701 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:46:15.327920 16778 external_mini_cluster.cc:1428] Setting key 00586b9e76df60cd3942e63c17f2a725
I20260807 22:46:15.328886 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34123
--builtin_ntp_servers=127.16.98.148:42465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260807 22:46:15.416543 26614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:46:15.416862 26614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:46:15.416899 26614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:46:15.418121 26614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:46:15.418184 26614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:46:15.418229 26614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:46:15.419801 26614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:46:15.420082 26614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:46:15.420506 26614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:46:15.426373 26622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:15.426461 26614 server_base.cc:1034] running on GCE node
W20260807 22:46:15.426390 26620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:15.426391 26619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:15.426838 26614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:46:15.427089 26614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:46:15.428262 26614 hybrid_clock.cc:648] HybridClock initialized: now 1786142775428242 us; error 36 us; skew 500 ppm
I20260807 22:46:15.430153 26614 webserver.cc:460] Webserver started at http://127.16.98.131:45463/ using document root <none> and password file <none>
I20260807 22:46:15.430454 26614 fs_manager.cc:359] Metadata directory not provided
I20260807 22:46:15.430514 26614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:46:15.430660 26614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:46:15.431825 26614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "aecd18e1070947fba59a9b46812eb75d"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "94f25f817a4ab4b8989139f47e57f25d"
server_key_iv: "e0806ab8aadf78cb09d6557f42de85dc"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.432214 26614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "aecd18e1070947fba59a9b46812eb75d"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "94f25f817a4ab4b8989139f47e57f25d"
server_key_iv: "e0806ab8aadf78cb09d6557f42de85dc"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.434098 26614 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260807 22:46:15.435110 26628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.435501 26614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:46:15.435653 26614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "aecd18e1070947fba59a9b46812eb75d"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "94f25f817a4ab4b8989139f47e57f25d"
server_key_iv: "e0806ab8aadf78cb09d6557f42de85dc"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.435767 26614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:46:15.453190 26614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:46:15.453540 26614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:46:15.453691 26614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:46:15.453943 26614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:46:15.454298 26614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:46:15.454350 26614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.454394 26614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:46:15.454423 26614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.461242 26614 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:39789
I20260807 22:46:15.461293 26743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:39789 every 8 connection(s)
I20260807 22:46:15.461656 26614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:46:15.465185 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 26614
I20260807 22:46:15.465302 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:46:15.465569 16778 external_mini_cluster.cc:1428] Setting key bed875ab50609e92b2bb13de547dd877
I20260807 22:46:15.467895 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:15.462496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:53367 (local address 127.16.98.190:34123)
0807 22:46:15.462590 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:15.462592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:15.462816 (+ 224us) server_negotiation.cc:407] Connection header received
0807 22:46:15.463358 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:15.463359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:15.463371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:15.463433 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:15.464165 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:15.464755 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:15.466182 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:15.466473 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:15.466558 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:15.467600 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:46:15.467602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:46:15.467603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:46:15.467608 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:46:15.467625 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:46:15.467630 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:46:15.467680 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:15.467766 (+ 86us) server_negotiation.cc:299] Negotiation successful
0807 22:46:15.467814 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 22:46:15.468117 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:15.462460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:34123 (local address 127.16.98.131:53367)
0807 22:46:15.462679 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:15.462715 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:15.463263 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:15.463505 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:15.463509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:15.463517 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:15.464007 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:15.464011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:15.464865 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:15.464866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:15.466076 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:15.466080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:15.466580 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:15.466581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:15.466756 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:15.467391 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:46:15.467391 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:15.467420 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:46:15.467444 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:46:15.467687 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:46:15.467694 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:46:15.467697 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:46:15.467766 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 22:46:15.467877 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":119,"threads_started":1}
I20260807 22:46:15.468691 26744 heartbeater.cc:347] Connected to a master server at 127.16.98.190:34123
I20260807 22:46:15.468814 26744 heartbeater.cc:464] Registering TS with master...
I20260807 22:46:15.469060 26744 heartbeater.cc:511] Master 127.16.98.190:34123 requested a full tablet report, sending...
I20260807 22:46:15.469534 26283 ts_manager.cc:194] Registered new tserver with Master: aecd18e1070947fba59a9b46812eb75d (127.16.98.131:39789)
I20260807 22:46:15.469934 26283 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.98.131:53367
I20260807 22:46:15.478215 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 22:46:15.480466 26323 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:15.479794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.16.98.190:34123)
0807 22:46:15.479893 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:15.479895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:15.479998 (+ 103us) server_negotiation.cc:407] Connection header received
0807 22:46:15.480135 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:15.480136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:15.480151 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:15.480209 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:15.480380 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50374: BlockingRecv error: recv got EOF from 127.0.0.1:50374 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 22:46:15.480499 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 26343
I20260807 22:46:15.487142 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 26480
I20260807 22:46:15.492183 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 26614
I20260807 22:46:15.498322 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 26252
2026-08-07T22:46:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (651 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-07T22:46:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:46:15Z Disabled control of system clock
I20260807 22:46:15.528774 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:36133
--webserver_interface=127.16.98.190
--webserver_port=0
--builtin_ntp_servers=127.16.98.148:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:36133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 22:46:15.614636 26755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:46:15.614869 26755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:46:15.614894 26755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:46:15.616044 26755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:46:15.616082 26755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:46:15.616096 26755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:46:15.616107 26755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:46:15.616120 26755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:46:15.617506 26755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:36133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:36133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:46:15.617722 26755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:46:15.618006 26755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:46:15.623567 26763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:15.623562 26761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:15.623653 26755 server_base.cc:1034] running on GCE node
W20260807 22:46:15.623698 26760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:15.624122 26755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:46:15.624450 26755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:46:15.625638 26755 hybrid_clock.cc:648] HybridClock initialized: now 1786142775625664 us; error 71 us; skew 500 ppm
I20260807 22:46:15.627049 26755 webserver.cc:460] Webserver started at http://127.16.98.190:35001/ using document root <none> and password file <none>
I20260807 22:46:15.627372 26755 fs_manager.cc:359] Metadata directory not provided
I20260807 22:46:15.627446 26755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:46:15.627601 26755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:46:15.628837 26755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "3671b61b8b0747b2aee8036714ccf81b"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "b3482323ca80a1e99cd46470c27b505c"
server_key_iv: "03e34a0fc2648d426bbe8e7c08d94a75"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.629209 26755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "3671b61b8b0747b2aee8036714ccf81b"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "b3482323ca80a1e99cd46470c27b505c"
server_key_iv: "03e34a0fc2648d426bbe8e7c08d94a75"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.630625 26755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:46:15.631667 26770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.631919 26755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:46:15.632082 26755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "3671b61b8b0747b2aee8036714ccf81b"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "b3482323ca80a1e99cd46470c27b505c"
server_key_iv: "03e34a0fc2648d426bbe8e7c08d94a75"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.632170 26755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:46:15.636175 26755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:46:15.636494 26755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:46:15.636682 26755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:46:15.641597 26755 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:36133
I20260807 22:46:15.641641 26823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:36133 every 8 connection(s)
I20260807 22:46:15.642030 26755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:46:15.642836 26824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:46:15.643733 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 26755
I20260807 22:46:15.643817 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:46:15.644049 16778 external_mini_cluster.cc:1428] Setting key 99620909e0aa8bc3b6fe4e5ae8517a76
I20260807 22:46:15.646961 26824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b: Bootstrap starting.
I20260807 22:46:15.647198 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:15.644717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.16.98.190:36133)
0807 22:46:15.644961 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:15.644971 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:15.645019 (+ 48us) server_negotiation.cc:407] Connection header received
0807 22:46:15.645668 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:15.645698 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:15.645755 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:15.645921 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:15.646709 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:46:15.646714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:46:15.646714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:46:15.646719 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:46:15.646771 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:46:15.646787 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:46:15.646867 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:15.646934 (+ 67us) server_negotiation.cc:299] Negotiation successful
0807 22:46:15.647002 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260807 22:46:15.647693 26824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b: Neither blocks nor log segments found. Creating new log.
I20260807 22:46:15.648017 26824 log.cc:824] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b: Log is configured to *not* fsync() on all Append() calls
I20260807 22:46:15.648833 26824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b: No bootstrap required, opened a new log
I20260807 22:46:15.650373 26824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3671b61b8b0747b2aee8036714ccf81b" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36133 } }
I20260807 22:46:15.650574 26824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:46:15.650664 26824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3671b61b8b0747b2aee8036714ccf81b, State: Initialized, Role: FOLLOWER
I20260807 22:46:15.650813 26824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3671b61b8b0747b2aee8036714ccf81b" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36133 } }
I20260807 22:46:15.650895 26824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:46:15.650921 26824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:46:15.650954 26824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:46:15.652062 26824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3671b61b8b0747b2aee8036714ccf81b" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36133 } }
I20260807 22:46:15.652163 26824 leader_election.cc:302] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3671b61b8b0747b2aee8036714ccf81b; no voters:
I20260807 22:46:15.652297 26824 leader_election.cc:288] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:46:15.652383 26829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:46:15.652534 26829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [term 1 LEADER]: Becoming Leader. State: Replica: 3671b61b8b0747b2aee8036714ccf81b, State: Running, Role: LEADER
I20260807 22:46:15.652638 26824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:46:15.652688 26829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3671b61b8b0747b2aee8036714ccf81b" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36133 } }
I20260807 22:46:15.653395 26830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3671b61b8b0747b2aee8036714ccf81b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3671b61b8b0747b2aee8036714ccf81b" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36133 } } }
I20260807 22:46:15.653491 26830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [sys.catalog]: This master's current role is: LEADER
I20260807 22:46:15.653314 26831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3671b61b8b0747b2aee8036714ccf81b. Latest consensus state: current_term: 1 leader_uuid: "3671b61b8b0747b2aee8036714ccf81b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3671b61b8b0747b2aee8036714ccf81b" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 36133 } } }
I20260807 22:46:15.653694 26831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b [sys.catalog]: This master's current role is: LEADER
I20260807 22:46:15.654075 26838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:46:15.654582 26838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:46:15.656235 26838 catalog_manager.cc:1347] Generated new cluster ID: f2b99a2b14dc45bda3c5a3302fa330c8
I20260807 22:46:15.656301 26838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:46:15.661649 26838 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:46:15.661818 26838 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:46:15.670885 26838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3671b61b8b0747b2aee8036714ccf81b: Generated new TSK 0
I20260807 22:46:15.671118 26838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:46:15.675864 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36133
--builtin_ntp_servers=127.16.98.148:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 22:46:15.771584 26848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:46:15.771837 26848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:46:15.771880 26848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:46:15.773272 26848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:46:15.773342 26848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:46:15.773393 26848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:46:15.774909 26848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:46:15.775203 26848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:46:15.775539 26848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:46:15.781867 26854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:15.781942 26853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:15.781867 26856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:15.782250 26848 server_base.cc:1034] running on GCE node
I20260807 22:46:15.782557 26848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:46:15.782804 26848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:46:15.783965 26848 hybrid_clock.cc:648] HybridClock initialized: now 1786142775783950 us; error 33 us; skew 500 ppm
I20260807 22:46:15.785413 26848 webserver.cc:460] Webserver started at http://127.16.98.129:37757/ using document root <none> and password file <none>
I20260807 22:46:15.785698 26848 fs_manager.cc:359] Metadata directory not provided
I20260807 22:46:15.785770 26848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:46:15.785917 26848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:46:15.786995 26848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "8a88956edb9a4f9eb97902e694d2bf84"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "3529dd55888439325d2e2c7d4b1487fb"
server_key_iv: "6b9b1fb2f62bf161d3afc1248a53353d"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.787354 26848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "8a88956edb9a4f9eb97902e694d2bf84"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "3529dd55888439325d2e2c7d4b1487fb"
server_key_iv: "6b9b1fb2f62bf161d3afc1248a53353d"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.788843 26848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:46:15.789830 26862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.790436 26848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:46:15.790562 26848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "8a88956edb9a4f9eb97902e694d2bf84"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "3529dd55888439325d2e2c7d4b1487fb"
server_key_iv: "6b9b1fb2f62bf161d3afc1248a53353d"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.790657 26848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:46:15.794291 26848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:46:15.794534 26848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:46:15.794634 26848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:46:15.794836 26848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:46:15.795184 26848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:46:15.795221 26848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.795241 26848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:46:15.795254 26848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.802711 26848 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:45275
I20260807 22:46:15.802790 26977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:45275 every 8 connection(s)
I20260807 22:46:15.803140 26848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:46:15.804011 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 26848
I20260807 22:46:15.804116 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:46:15.804320 16778 external_mini_cluster.cc:1428] Setting key 1f03f77fa2ae131877040657613eadd1
I20260807 22:46:15.805459 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36133
--builtin_ntp_servers=127.16.98.148:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 22:46:15.806878 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:15.803883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:43635 (local address 127.16.98.190:36133)
0807 22:46:15.803931 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:15.803934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:15.804302 (+ 368us) server_negotiation.cc:407] Connection header received
0807 22:46:15.805016 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:15.805017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:15.805023 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:15.805071 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:15.806603 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:46:15.806604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:46:15.806604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:46:15.806609 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:46:15.806625 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:46:15.806629 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:46:15.806692 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:15.806810 (+ 118us) server_negotiation.cc:299] Negotiation successful
0807 22:46:15.806825 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 22:46:15.807159 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:15.803908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36133 (local address 127.16.98.129:43635)
0807 22:46:15.804161 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:15.804203 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:15.804890 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:15.805140 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:15.805144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:15.805153 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:15.806383 (+ 1230us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:46:15.806384 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:15.806435 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:46:15.806469 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:46:15.806706 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:46:15.806712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:46:15.806715 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:46:15.806793 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 22:46:15.806860 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260807 22:46:15.807627 26978 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36133
I20260807 22:46:15.807734 26978 heartbeater.cc:464] Registering TS with master...
I20260807 22:46:15.807890 26978 heartbeater.cc:511] Master 127.16.98.190:36133 requested a full tablet report, sending...
I20260807 22:46:15.808386 26787 ts_manager.cc:194] Registered new tserver with Master: 8a88956edb9a4f9eb97902e694d2bf84 (127.16.98.129:45275)
W20260807 22:46:15.893426 26981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:46:15.893679 26981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:46:15.893704 26981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:46:15.895027 26981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:46:15.895074 26981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:46:15.895102 26981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:46:15.896587 26981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:46:15.896791 26981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:46:15.897047 26981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:46:15.903165 26989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:15.903169 26986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:15.903169 26987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:15.903206 26981 server_base.cc:1034] running on GCE node
I20260807 22:46:15.903579 26981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:46:15.903798 26981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:46:15.904948 26981 hybrid_clock.cc:648] HybridClock initialized: now 1786142775904925 us; error 37 us; skew 500 ppm
I20260807 22:46:15.906453 26981 webserver.cc:460] Webserver started at http://127.16.98.130:37121/ using document root <none> and password file <none>
I20260807 22:46:15.906733 26981 fs_manager.cc:359] Metadata directory not provided
I20260807 22:46:15.906807 26981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:46:15.906958 26981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:46:15.908087 26981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "0e4d3f1f809a4b1197118460fc9e4ed3"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "7f085c04debc6dd259db0aa917d00a50"
server_key_iv: "28f22d9408229d4b8731290a0874e0b7"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.908465 26981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "0e4d3f1f809a4b1197118460fc9e4ed3"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "7f085c04debc6dd259db0aa917d00a50"
server_key_iv: "28f22d9408229d4b8731290a0874e0b7"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.909921 26981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:46:15.910836 26995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.911109 26981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:46:15.911221 26981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "0e4d3f1f809a4b1197118460fc9e4ed3"
format_stamp: "Formatted at 2026-08-07 22:46:15 on dist-test-slave-rgb1"
server_key: "7f085c04debc6dd259db0aa917d00a50"
server_key_iv: "28f22d9408229d4b8731290a0874e0b7"
server_key_version: "encryptionkey@0"
I20260807 22:46:15.911310 26981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:46:15.914930 26981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:46:15.915222 26981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:46:15.915369 26981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:46:15.915576 26981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:46:15.915910 26981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:46:15.915966 26981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.916016 26981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:46:15.916045 26981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:15.922485 26981 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:39915
I20260807 22:46:15.922544 27110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:39915 every 8 connection(s)
I20260807 22:46:15.922919 26981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:46:15.926501 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:15.924092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:53143 (local address 127.16.98.190:36133)
0807 22:46:15.924183 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:15.924186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:15.924198 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:46:15.924605 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:15.924607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:15.924614 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:15.924661 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:15.926180 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:46:15.926183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:46:15.926183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:46:15.926189 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:46:15.926209 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:46:15.926215 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:46:15.926280 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:15.926389 (+ 109us) server_negotiation.cc:299] Negotiation successful
0807 22:46:15.926409 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 22:46:15.926760 27113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:15.923699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36133 (local address 127.16.98.130:53143)
0807 22:46:15.923891 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:15.923936 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:15.924451 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:15.924734 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:15.924738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:15.924746 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:15.925916 (+ 1170us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:46:15.925917 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:15.925950 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:46:15.926045 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:46:15.926285 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:46:15.926291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:46:15.926295 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:46:15.926402 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 22:46:15.926477 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260807 22:46:15.927570 27111 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36133
I20260807 22:46:15.927671 27111 heartbeater.cc:464] Registering TS with master...
I20260807 22:46:15.927805 27111 heartbeater.cc:511] Master 127.16.98.190:36133 requested a full tablet report, sending...
I20260807 22:46:15.928220 26787 ts_manager.cc:194] Registered new tserver with Master: 0e4d3f1f809a4b1197118460fc9e4ed3 (127.16.98.130:39915)
I20260807 22:46:15.932175 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 26981
I20260807 22:46:15.932319 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:46:15.932610 16778 external_mini_cluster.cc:1428] Setting key 5522762ef49647f873f120833dfa207a
I20260807 22:46:15.933943 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36133
--builtin_ntp_servers=127.16.98.148:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 22:46:16.022655 27114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:46:16.022892 27114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:46:16.022930 27114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:46:16.024087 27114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:46:16.024151 27114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:46:16.024212 27114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:46:16.025660 27114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:36133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:46:16.025925 27114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:46:16.026228 27114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:46:16.031374 27120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:16.031397 27119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:16.031374 27122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:16.031672 27114 server_base.cc:1034] running on GCE node
I20260807 22:46:16.031884 27114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:46:16.032208 27114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:46:16.033366 27114 hybrid_clock.cc:648] HybridClock initialized: now 1786142776033359 us; error 36 us; skew 500 ppm
I20260807 22:46:16.034636 27114 webserver.cc:460] Webserver started at http://127.16.98.131:39189/ using document root <none> and password file <none>
I20260807 22:46:16.035050 27114 fs_manager.cc:359] Metadata directory not provided
I20260807 22:46:16.035130 27114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:46:16.035313 27114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:46:16.036445 27114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "810c00edd1d84512a4d7de9f422dd58b"
format_stamp: "Formatted at 2026-08-07 22:46:16 on dist-test-slave-rgb1"
server_key: "52fcca94f9c4654560deb9e09757bd65"
server_key_iv: "454ea1ee2457ef17720d17c0937bacf7"
server_key_version: "encryptionkey@0"
I20260807 22:46:16.036895 27114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "810c00edd1d84512a4d7de9f422dd58b"
format_stamp: "Formatted at 2026-08-07 22:46:16 on dist-test-slave-rgb1"
server_key: "52fcca94f9c4654560deb9e09757bd65"
server_key_iv: "454ea1ee2457ef17720d17c0937bacf7"
server_key_version: "encryptionkey@0"
I20260807 22:46:16.038345 27114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:46:16.039310 27128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:16.039554 27114 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:46:16.039650 27114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "810c00edd1d84512a4d7de9f422dd58b"
format_stamp: "Formatted at 2026-08-07 22:46:16 on dist-test-slave-rgb1"
server_key: "52fcca94f9c4654560deb9e09757bd65"
server_key_iv: "454ea1ee2457ef17720d17c0937bacf7"
server_key_version: "encryptionkey@0"
I20260807 22:46:16.039731 27114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:46:16.043444 27114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:46:16.043725 27114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:46:16.043865 27114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:46:16.044095 27114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:46:16.044412 27114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:46:16.044467 27114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:16.044508 27114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:46:16.044535 27114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:16.051486 27114 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:32983
I20260807 22:46:16.051681 27244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:32983 every 8 connection(s)
I20260807 22:46:16.051962 27114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:46:16.055606 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:16.053003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:43063 (local address 127.16.98.190:36133)
0807 22:46:16.053083 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:16.053085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:16.053341 (+ 256us) server_negotiation.cc:407] Connection header received
0807 22:46:16.053972 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:16.053974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:16.053982 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:16.054046 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:16.055337 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:46:16.055338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:46:16.055339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:46:16.055344 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:46:16.055359 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:46:16.055363 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:46:16.055417 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:16.055516 (+ 99us) server_negotiation.cc:299] Negotiation successful
0807 22:46:16.055533 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 22:46:16.055832 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:16.052938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:36133 (local address 127.16.98.131:43063)
0807 22:46:16.053193 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:16.053239 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:16.053819 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:16.054138 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:16.054143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:16.054159 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:16.055172 (+ 1013us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:46:16.055173 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:16.055203 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:46:16.055230 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:46:16.055423 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:46:16.055428 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:46:16.055431 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:46:16.055520 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 22:46:16.055584 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
I20260807 22:46:16.056340 27245 heartbeater.cc:347] Connected to a master server at 127.16.98.190:36133
I20260807 22:46:16.056464 27245 heartbeater.cc:464] Registering TS with master...
I20260807 22:46:16.056682 27245 heartbeater.cc:511] Master 127.16.98.190:36133 requested a full tablet report, sending...
I20260807 22:46:16.057085 26787 ts_manager.cc:194] Registered new tserver with Master: 810c00edd1d84512a4d7de9f422dd58b (127.16.98.131:32983)
I20260807 22:46:16.060812 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 27114
I20260807 22:46:16.060912 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:46:16.061137 16778 external_mini_cluster.cc:1428] Setting key 78d6e0bed3ee4f6f4af493cabd7d974f
I20260807 22:46:16.062191 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 22:46:16.064239 26827 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:46:16.063655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.16.98.190:36133)
0807 22:46:16.063731 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:16.063733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:16.063885 (+ 152us) server_negotiation.cc:407] Connection header received
0807 22:46:16.063911 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:16.063912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:16.063917 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:16.063974 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:16.064154 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48826: BlockingRecv error: recv got EOF from 127.0.0.1:48826 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 22:46:16.064306 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 26848
I20260807 22:46:16.071430 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 26981
I20260807 22:46:16.077072 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 27114
I20260807 22:46:16.082275 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 26755
2026-08-07T22:46:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (583 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:46:16 dist-test-slave-rgb1 krb5kdc[27256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:46:16 dist-test-slave-rgb1 krb5kdc[27256](info): set up 2 sockets
Aug 07 22:46:16 dist-test-slave-rgb1 krb5kdc[27256](info): commencing operation
krb5kdc: starting...
W20260807 22:46:17.220899 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:46:17 dist-test-slave-rgb1 krb5kdc[27256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:46:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:46:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:46:17.368976 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:44565
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:44565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:46:17.464073 27270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:46:17.464413 27270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:46:17.464458 27270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:46:17.465859 27270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:46:17.465935 27270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:46:17.465965 27270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:46:17.465997 27270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:46:17.466032 27270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:46:17.467955 27270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:44565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:44565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:46:17.468300 27270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:46:17.468604 27270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:46:17.474427 27276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:17.474387 27278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:17.474447 27275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:17.474579 27270 server_base.cc:1034] running on GCE node
I20260807 22:46:17.474895 27270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:46:17.475271 27270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:46:17.476470 27270 hybrid_clock.cc:648] HybridClock initialized: now 1786142777476455 us; error 49 us; skew 500 ppm
Aug 07 22:46:17 dist-test-slave-rgb1 krb5kdc[27256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:46:17.479470 27270 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:46:17.480044 27270 webserver.cc:460] Webserver started at http://127.16.98.190:32935/ using document root <none> and password file <none>
I20260807 22:46:17.480306 27270 fs_manager.cc:359] Metadata directory not provided
I20260807 22:46:17.480381 27270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:46:17.480545 27270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:46:17.481820 27270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "fb5a5b09f00943ae8bad9c7d6d2910a6"
format_stamp: "Formatted at 2026-08-07 22:46:17 on dist-test-slave-rgb1"
server_key: "dea40c7d19eb6ae5f94740c635550f0d"
server_key_iv: "65a56d4ba99af224ee235c4e9d61201e"
server_key_version: "encryptionkey@0"
I20260807 22:46:17.482235 27270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "fb5a5b09f00943ae8bad9c7d6d2910a6"
format_stamp: "Formatted at 2026-08-07 22:46:17 on dist-test-slave-rgb1"
server_key: "dea40c7d19eb6ae5f94740c635550f0d"
server_key_iv: "65a56d4ba99af224ee235c4e9d61201e"
server_key_version: "encryptionkey@0"
I20260807 22:46:17.483765 27270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:46:17.484668 27286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:17.484906 27270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:46:17.485029 27270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "fb5a5b09f00943ae8bad9c7d6d2910a6"
format_stamp: "Formatted at 2026-08-07 22:46:17 on dist-test-slave-rgb1"
server_key: "dea40c7d19eb6ae5f94740c635550f0d"
server_key_iv: "65a56d4ba99af224ee235c4e9d61201e"
server_key_version: "encryptionkey@0"
I20260807 22:46:17.485123 27270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:46:17.509413 27270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:46:17.512132 27270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:46:17.512346 27270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:46:17.517221 27270 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:44565
I20260807 22:46:17.517294 27344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:44565 every 8 connection(s)
I20260807 22:46:17.517705 27270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:46:17.518853 27345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:46:17.521347 27345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6: Bootstrap starting.
I20260807 22:46:17.521971 27345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6: Neither blocks nor log segments found. Creating new log.
I20260807 22:46:17.522300 27345 log.cc:824] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6: Log is configured to *not* fsync() on all Append() calls
I20260807 22:46:17.523145 27345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6: No bootstrap required, opened a new log
I20260807 22:46:17.524835 27345 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5a5b09f00943ae8bad9c7d6d2910a6" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44565 } }
I20260807 22:46:17.524976 27345 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:46:17.525000 27345 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5a5b09f00943ae8bad9c7d6d2910a6, State: Initialized, Role: FOLLOWER
I20260807 22:46:17.525192 27345 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5a5b09f00943ae8bad9c7d6d2910a6" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44565 } }
I20260807 22:46:17.525259 27345 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:46:17.525295 27345 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:46:17.525346 27345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:46:17.526198 27345 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5a5b09f00943ae8bad9c7d6d2910a6" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44565 } }
I20260807 22:46:17.526317 27345 leader_election.cc:302] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb5a5b09f00943ae8bad9c7d6d2910a6; no voters:
I20260807 22:46:17.526512 27345 leader_election.cc:288] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:46:17.526510 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 27270
I20260807 22:46:17.526549 27348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:46:17.526649 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:46:17.526665 27348 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [term 1 LEADER]: Becoming Leader. State: Replica: fb5a5b09f00943ae8bad9c7d6d2910a6, State: Running, Role: LEADER
I20260807 22:46:17.526739 27348 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5a5b09f00943ae8bad9c7d6d2910a6" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44565 } }
I20260807 22:46:17.526796 27345 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:46:17.526895 16778 external_mini_cluster.cc:1428] Setting key f48e265733c140cfd36d6aec1f7f2527
I20260807 22:46:17.527040 27348 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb5a5b09f00943ae8bad9c7d6d2910a6. Latest consensus state: current_term: 1 leader_uuid: "fb5a5b09f00943ae8bad9c7d6d2910a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5a5b09f00943ae8bad9c7d6d2910a6" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44565 } } }
I20260807 22:46:17.527115 27348 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [sys.catalog]: This master's current role is: LEADER
I20260807 22:46:17.527069 27349 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb5a5b09f00943ae8bad9c7d6d2910a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5a5b09f00943ae8bad9c7d6d2910a6" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44565 } } }
I20260807 22:46:17.527360 27349 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6 [sys.catalog]: This master's current role is: LEADER
I20260807 22:46:17.527576 27353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:46:17.528292 27353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:46:17.530215 27353 catalog_manager.cc:1347] Generated new cluster ID: b8f1937ceb234583bc00af3694c188d2
I20260807 22:46:17.530277 27353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 22:46:17 dist-test-slave-rgb1 krb5kdc[27256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:46:17.537700 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:17.527432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.16.98.190:44565)
0807 22:46:17.527606 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:17.527617 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:17.527875 (+ 258us) server_negotiation.cc:407] Connection header received
0807 22:46:17.528009 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:17.528017 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:17.528097 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:17.528231 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:17.529266 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:17.530482 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:17.531212 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:17.531643 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:17.531762 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:17.534175 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:46:17.534178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:46:17.534179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:46:17.534190 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:46:17.536039 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:46:17.536431 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:46:17.536435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:46:17.536435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:46:17.536548 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:46:17.536792 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:46:17.536793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:46:17.536793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:46:17.537055 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:46:17.537233 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:17.537305 (+ 72us) server_negotiation.cc:299] Negotiation successful
0807 22:46:17.537457 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260807 22:46:17.542693 27353 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:46:17.543282 27353 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:46:17.545657 27353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6: Generated new TSK 0
I20260807 22:46:17.545845 27353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:46:17.588675 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:44565
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:46:17.704723 27372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:46:17.704998 27372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:46:17.705047 27372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:46:17.706413 27372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:46:17.706487 27372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:46:17.706547 27372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:46:17.708410 27372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:44565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:46:17.708778 27372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:46:17.709121 27372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:46:17.715968 27381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:17.716048 27372 server_base.cc:1034] running on GCE node
W20260807 22:46:17.716174 27378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:17.716184 27379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:17.716578 27372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:46:17.716882 27372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:46:17.718063 27372 hybrid_clock.cc:648] HybridClock initialized: now 1786142777718049 us; error 31 us; skew 500 ppm
Aug 07 22:46:17 dist-test-slave-rgb1 krb5kdc[27256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:46:17.721191 27372 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:46:17.721706 27372 webserver.cc:460] Webserver started at http://127.16.98.129:35359/ using document root <none> and password file <none>
I20260807 22:46:17.721992 27372 fs_manager.cc:359] Metadata directory not provided
I20260807 22:46:17.722044 27372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:46:17.722149 27372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:46:17.723347 27372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "469c9a966f7b4fb088f161f47f7e316e"
format_stamp: "Formatted at 2026-08-07 22:46:17 on dist-test-slave-rgb1"
server_key: "abeb1f6ce75e7d2a1c99dc52ff80e86e"
server_key_iv: "d3563b76582e445c41ce0e6e883b140b"
server_key_version: "encryptionkey@0"
I20260807 22:46:17.723747 27372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "469c9a966f7b4fb088f161f47f7e316e"
format_stamp: "Formatted at 2026-08-07 22:46:17 on dist-test-slave-rgb1"
server_key: "abeb1f6ce75e7d2a1c99dc52ff80e86e"
server_key_iv: "d3563b76582e445c41ce0e6e883b140b"
server_key_version: "encryptionkey@0"
I20260807 22:46:17.725322 27372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:46:17.726262 27388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:17.726588 27372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:46:17.726691 27372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "469c9a966f7b4fb088f161f47f7e316e"
format_stamp: "Formatted at 2026-08-07 22:46:17 on dist-test-slave-rgb1"
server_key: "abeb1f6ce75e7d2a1c99dc52ff80e86e"
server_key_iv: "d3563b76582e445c41ce0e6e883b140b"
server_key_version: "encryptionkey@0"
I20260807 22:46:17.726799 27372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:46:17.754215 27372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:46:17.757067 27372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:46:17.757246 27372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:46:17.757531 27372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:46:17.757885 27372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:46:17.757930 27372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:17.757975 27372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:46:17.758005 27372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:17.765021 27372 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:41895
I20260807 22:46:17.765103 27503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:41895 every 8 connection(s)
I20260807 22:46:17.765488 27372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:46:17.766287 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 27372
I20260807 22:46:17.766409 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:46:17.766630 16778 external_mini_cluster.cc:1428] Setting key 81c13546cd74570036b3f678d5aac244
Aug 07 22:46:17 dist-test-slave-rgb1 krb5kdc[27256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
I20260807 22:46:17.777393 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:17.766569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:35645 (local address 127.16.98.190:44565)
0807 22:46:17.766644 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:17.766647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:17.767123 (+ 476us) server_negotiation.cc:407] Connection header received
0807 22:46:17.767737 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:17.767738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:17.767751 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:17.767819 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:17.768904 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:17.769483 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:17.770694 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:17.770952 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:17.771150 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:17.773885 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:46:17.773886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:46:17.773887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:46:17.773896 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:46:17.775814 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:46:17.776325 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:46:17.776326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:46:17.776326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:46:17.776391 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:46:17.776684 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:46:17.776685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:46:17.776685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:46:17.776812 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:46:17.776903 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:17.777200 (+ 297us) server_negotiation.cc:299] Negotiation successful
0807 22:46:17.777299 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 22:46:17.777595 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:17.766420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:44565 (local address 127.16.98.129:35645)
0807 22:46:17.766932 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:17.766982 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:17.767640 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:17.767894 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:17.767898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:17.768502 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:17.768782 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:17.768786 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:17.769617 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:17.769619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:17.770585 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:17.770588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:17.771175 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:17.771177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:17.771317 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:17.771883 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:17.771891 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:17.773781 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:46:17.775987 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:46:17.775996 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:46:17.775998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:46:17.776242 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:46:17.776468 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:46:17.776469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:46:17.776469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:46:17.776621 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:46:17.776896 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:46:17.776904 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:46:17.777046 (+ 142us) client_negotiation.cc:769] Sending connection context
0807 22:46:17.777163 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 22:46:17.777338 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":97,"threads_started":1}
I20260807 22:46:17.778170 27504 heartbeater.cc:347] Connected to a master server at 127.16.98.190:44565
I20260807 22:46:17.778294 27504 heartbeater.cc:464] Registering TS with master...
I20260807 22:46:17.778505 27504 heartbeater.cc:511] Master 127.16.98.190:44565 requested a full tablet report, sending...
I20260807 22:46:17.779104 27304 ts_manager.cc:194] Registered new tserver with Master: 469c9a966f7b4fb088f161f47f7e316e (127.16.98.129:41895)
I20260807 22:46:17.779817 27304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:35645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:46:17.799961 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:44565
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:46:17.890812 27509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:46:17.891086 27509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:46:17.891119 27509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:46:17.892390 27509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:46:17.892442 27509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:46:17.892472 27509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:46:17.894104 27509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:44565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:46:17.894354 27509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:46:17.894565 27509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:46:17.899946 27515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:17.900002 27514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:17.900063 27517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:17.900215 27509 server_base.cc:1034] running on GCE node
I20260807 22:46:17.900465 27509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:46:17.900796 27509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:46:17.901978 27509 hybrid_clock.cc:648] HybridClock initialized: now 1786142777901923 us; error 78 us; skew 500 ppm
Aug 07 22:46:17 dist-test-slave-rgb1 krb5kdc[27256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:46:17.904803 27509 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:46:17.905304 27509 webserver.cc:460] Webserver started at http://127.16.98.130:41747/ using document root <none> and password file <none>
I20260807 22:46:17.905555 27509 fs_manager.cc:359] Metadata directory not provided
I20260807 22:46:17.905623 27509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:46:17.905767 27509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:46:17.906847 27509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "3e84edfe9ce641a580ccaab3b8b6970b"
format_stamp: "Formatted at 2026-08-07 22:46:17 on dist-test-slave-rgb1"
server_key: "8eb0a5b081caf159bb0dbf9a84d201f7"
server_key_iv: "650564cd720d700f02473a4b0b8573b5"
server_key_version: "encryptionkey@0"
I20260807 22:46:17.907251 27509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "3e84edfe9ce641a580ccaab3b8b6970b"
format_stamp: "Formatted at 2026-08-07 22:46:17 on dist-test-slave-rgb1"
server_key: "8eb0a5b081caf159bb0dbf9a84d201f7"
server_key_iv: "650564cd720d700f02473a4b0b8573b5"
server_key_version: "encryptionkey@0"
I20260807 22:46:17.908797 27509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:46:17.909807 27524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:17.910094 27509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:46:17.910221 27509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "3e84edfe9ce641a580ccaab3b8b6970b"
format_stamp: "Formatted at 2026-08-07 22:46:17 on dist-test-slave-rgb1"
server_key: "8eb0a5b081caf159bb0dbf9a84d201f7"
server_key_iv: "650564cd720d700f02473a4b0b8573b5"
server_key_version: "encryptionkey@0"
I20260807 22:46:17.910307 27509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:46:17.924216 27509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:46:17.926905 27509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:46:17.927129 27509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:46:17.927412 27509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:46:17.927763 27509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:46:17.927819 27509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:17.927860 27509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:46:17.927902 27509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:17.935400 27509 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:40303
I20260807 22:46:17.935467 27640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:40303 every 8 connection(s)
I20260807 22:46:17.935835 27509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:46:17.936398 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 27509
I20260807 22:46:17.936532 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:46:17.936820 16778 external_mini_cluster.cc:1428] Setting key a49a8f9aabe0db73912795b0aef82bdd
Aug 07 22:46:17 dist-test-slave-rgb1 krb5kdc[27256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142777, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
I20260807 22:46:17.947938 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:17.936676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:32969 (local address 127.16.98.190:44565)
0807 22:46:17.936763 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:17.936765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:17.937061 (+ 296us) server_negotiation.cc:407] Connection header received
0807 22:46:17.937773 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:17.937774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:17.937792 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:17.937848 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:17.939451 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:17.940130 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:17.941345 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:17.941574 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:17.941671 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:17.944156 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:46:17.944159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:46:17.944160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:46:17.944170 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:46:17.946154 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:46:17.946694 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:46:17.946696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:46:17.946696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:46:17.946765 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:46:17.947156 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:46:17.947157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:46:17.947158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:46:17.947343 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:46:17.947427 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:17.947782 (+ 355us) server_negotiation.cc:299] Negotiation successful
0807 22:46:17.947865 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 22:46:17.948341 27643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:17.936665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:44565 (local address 127.16.98.130:32969)
0807 22:46:17.936879 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:17.936943 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:17.937646 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:17.937935 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:17.937941 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:17.938952 (+ 1011us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:17.939274 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:17.939286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:17.940247 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:17.940249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:17.941236 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:17.941239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:17.941683 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:17.941684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:17.941827 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:17.942475 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:17.942485 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:17.944048 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:46:17.946258 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:46:17.946267 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:46:17.946269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:46:17.946599 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:46:17.946876 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:46:17.946936 (+ 60us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:46:17.946936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:46:17.947068 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:46:17.947483 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:46:17.947491 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:46:17.947637 (+ 146us) client_negotiation.cc:769] Sending connection context
0807 22:46:17.947742 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 22:46:17.947886 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260807 22:46:17.948909 27641 heartbeater.cc:347] Connected to a master server at 127.16.98.190:44565
I20260807 22:46:17.949036 27641 heartbeater.cc:464] Registering TS with master...
I20260807 22:46:17.949239 27641 heartbeater.cc:511] Master 127.16.98.190:44565 requested a full tablet report, sending...
I20260807 22:46:17.950006 27304 ts_manager.cc:194] Registered new tserver with Master: 3e84edfe9ce641a580ccaab3b8b6970b (127.16.98.130:40303)
I20260807 22:46:17.950443 27304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:32969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:46:17.972335 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:44565
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:46:18.070861 27646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:46:18.071137 27646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:46:18.071166 27646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:46:18.072429 27646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:46:18.072474 27646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:46:18.072502 27646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:46:18.074244 27646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:44565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:46:18.074496 27646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:46:18.074707 27646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:46:18.080547 27652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:46:18.080547 27653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:18.080651 27646 server_base.cc:1034] running on GCE node
W20260807 22:46:18.080547 27655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:46:18.081063 27646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:46:18.081308 27646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:46:18.082468 27646 hybrid_clock.cc:648] HybridClock initialized: now 1786142778082438 us; error 49 us; skew 500 ppm
Aug 07 22:46:18 dist-test-slave-rgb1 krb5kdc[27256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142778, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:46:18.085433 27646 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:46:18.085963 27646 webserver.cc:460] Webserver started at http://127.16.98.131:33391/ using document root <none> and password file <none>
I20260807 22:46:18.086254 27646 fs_manager.cc:359] Metadata directory not provided
I20260807 22:46:18.086308 27646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:46:18.086444 27646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:46:18.087643 27646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "9191bf84a4c14e0fb72406bb797cc465"
format_stamp: "Formatted at 2026-08-07 22:46:18 on dist-test-slave-rgb1"
server_key: "1eeff6cf75509f54e8971cd6b8014715"
server_key_iv: "878848bdac7ad154d7c3e7ebb4aa48b1"
server_key_version: "encryptionkey@0"
I20260807 22:46:18.088073 27646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "9191bf84a4c14e0fb72406bb797cc465"
format_stamp: "Formatted at 2026-08-07 22:46:18 on dist-test-slave-rgb1"
server_key: "1eeff6cf75509f54e8971cd6b8014715"
server_key_iv: "878848bdac7ad154d7c3e7ebb4aa48b1"
server_key_version: "encryptionkey@0"
I20260807 22:46:18.089723 27646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:46:18.090672 27663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:18.090996 27646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:46:18.091095 27646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "9191bf84a4c14e0fb72406bb797cc465"
format_stamp: "Formatted at 2026-08-07 22:46:18 on dist-test-slave-rgb1"
server_key: "1eeff6cf75509f54e8971cd6b8014715"
server_key_iv: "878848bdac7ad154d7c3e7ebb4aa48b1"
server_key_version: "encryptionkey@0"
I20260807 22:46:18.091217 27646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:46:18.103998 27646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:46:18.106730 27646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:46:18.106926 27646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:46:18.107223 27646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:46:18.107584 27646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:46:18.107625 27646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:18.107650 27646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:46:18.107664 27646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:46:18.114725 27646 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:34019
I20260807 22:46:18.114792 27781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:34019 every 8 connection(s)
I20260807 22:46:18.115164 27646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:46:18.120078 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 27646
I20260807 22:46:18.120216 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:46:18.120424 16778 external_mini_cluster.cc:1428] Setting key 34c5dce55f7ab57ec2bd36fc922b6d3f
Aug 07 22:46:18 dist-test-slave-rgb1 krb5kdc[27256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142778, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:46:18.126448 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:18.116056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:39361 (local address 127.16.98.190:44565)
0807 22:46:18.116135 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:18.116138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:18.116407 (+ 269us) server_negotiation.cc:407] Connection header received
0807 22:46:18.117134 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:18.117135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:18.117149 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:18.117217 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:18.118391 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.119023 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.120196 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.120536 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.120666 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:18.123304 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:46:18.123306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:46:18.123306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:46:18.123314 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:46:18.125089 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:46:18.125563 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:46:18.125564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:46:18.125565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:46:18.125623 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:46:18.125860 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:46:18.125861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:46:18.125861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:46:18.125980 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:46:18.126039 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:18.126301 (+ 262us) server_negotiation.cc:299] Negotiation successful
0807 22:46:18.126375 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 22:46:18.126775 27784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:18.116029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:44565 (local address 127.16.98.131:39361)
0807 22:46:18.116267 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:18.116310 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:18.116993 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:18.117342 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:18.117347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:18.118051 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:46:18.118281 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:18.118286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.119126 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.119128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:18.120077 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:18.120080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.120703 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.120705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:18.120890 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:18.121671 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:46:18.121680 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:46:18.123205 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:46:18.125186 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:46:18.125194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:46:18.125196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:46:18.125459 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:46:18.125686 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:46:18.125687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:46:18.125687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:46:18.125801 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:46:18.126040 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:46:18.126045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:46:18.126176 (+ 131us) client_negotiation.cc:769] Sending connection context
0807 22:46:18.126269 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 22:46:18.126407 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":114,"threads_started":1}
I20260807 22:46:18.127439 27782 heartbeater.cc:347] Connected to a master server at 127.16.98.190:44565
I20260807 22:46:18.127559 27782 heartbeater.cc:464] Registering TS with master...
I20260807 22:46:18.127779 27782 heartbeater.cc:511] Master 127.16.98.190:44565 requested a full tablet report, sending...
I20260807 22:46:18.128327 27304 ts_manager.cc:194] Registered new tserver with Master: 9191bf84a4c14e0fb72406bb797cc465 (127.16.98.131:34019)
I20260807 22:46:18.128865 27304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:39361
I20260807 22:46:18.132882 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:46:18.141448 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:18.134374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.16.98.190:44565)
0807 22:46:18.134446 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:18.134449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:18.134580 (+ 131us) server_negotiation.cc:407] Connection header received
0807 22:46:18.134640 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:18.134641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:18.134652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:18.134705 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:46:18.135317 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.136030 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.136960 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.137164 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.137236 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:18.138560 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:46:18.138561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:46:18.138562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:46:18.138569 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:46:18.140154 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:46:18.140513 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:46:18.140514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:46:18.140515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:46:18.140627 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:46:18.140818 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:46:18.140819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:46:18.140819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:46:18.140978 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:46:18.141112 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:18.141267 (+ 155us) server_negotiation.cc:299] Negotiation successful
0807 22:46:18.141336 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 22:46:18.143561 27304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:46:18.145524 27304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:46:18.153650 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:18.148510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:41895 (local address 127.0.0.1:33750)
0807 22:46:18.148881 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:18.148907 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:18.148957 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:18.149535 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:18.149539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:18.149541 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:46:18.149624 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:18.149626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.151165 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.151167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:18.152352 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:18.152355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.153244 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.153245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:18.153354 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:18.153374 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 22:46:18.153452 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 22:46:18.153530 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":160,"threads_started":1}
I20260807 22:46:18.153690 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:18.148999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.16.98.130:40303)
0807 22:46:18.149382 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:18.149387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:18.149401 (+ 14us) server_negotiation.cc:407] Connection header received
0807 22:46:18.149446 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:18.149449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:18.149468 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:18.149575 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:46:18.149837 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.150711 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.152342 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.153192 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.153274 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:18.153338 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:18.153545 (+ 207us) server_negotiation.cc:299] Negotiation successful
0807 22:46:18.153592 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":131,"threads_started":1}
I20260807 22:46:18.153864 27795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:18.148590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60790 (local address 127.16.98.131:34019)
0807 22:46:18.149671 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:18.149676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:18.149686 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:46:18.149728 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:18.149730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:18.149747 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:18.149834 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:46:18.149973 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.151112 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.152524 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.153305 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.153393 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:18.153449 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:18.153706 (+ 257us) server_negotiation.cc:299] Negotiation successful
0807 22:46:18.153764 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1040,"thread_start_us":124,"threads_started":1}
I20260807 22:46:18.153995 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:18.148455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:34019 (local address 127.0.0.1:60790)
0807 22:46:18.148756 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:18.148806 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:18.148919 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:18.149843 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:18.149844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:18.149847 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:46:18.149887 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:18.149889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.151220 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.151221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:18.152452 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:18.152455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.153394 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.153395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:18.153467 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:18.153470 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:46:18.153523 (+ 53us) client_negotiation.cc:240] Negotiation successful
0807 22:46:18.153601 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":51,"threads_started":1}
I20260807 22:46:18.154172 27796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:18.148709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:40303 (local address 127.0.0.1:49344)
0807 22:46:18.148979 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:18.149107 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:18.149185 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:18.149663 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:18.149664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:18.149667 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:46:18.149720 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:18.149722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.150826 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.150827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:18.152253 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:18.152256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.153381 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.153383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:18.153456 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:18.153462 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 22:46:18.153525 (+ 63us) client_negotiation.cc:240] Negotiation successful
0807 22:46:18.153578 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":70,"threads_started":1}
I20260807 22:46:18.154470 27797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:18.148684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.16.98.129:41895)
0807 22:46:18.149336 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:18.149341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:18.149352 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:46:18.149400 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:18.149402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:18.149416 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:18.149518 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:46:18.149729 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.151026 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.152453 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.153129 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.153241 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:18.153314 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:18.154321 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0807 22:46:18.154378 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":103,"threads_started":1}
I20260807 22:46:18.154901 27715 tablet_service.cc:1467] Processing CreateTablet for tablet 97e1fe88ea8846eca38a1c2baa4e7b42 (DEFAULT_TABLE table=test-table [id=7124e1c7a93c4b3e988bdaa558f8707f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:46:18.155321 27715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97e1fe88ea8846eca38a1c2baa4e7b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:46:18.155262 27437 tablet_service.cc:1467] Processing CreateTablet for tablet 97e1fe88ea8846eca38a1c2baa4e7b42 (DEFAULT_TABLE table=test-table [id=7124e1c7a93c4b3e988bdaa558f8707f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:46:18.155606 27437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97e1fe88ea8846eca38a1c2baa4e7b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:46:18.157425 27573 tablet_service.cc:1467] Processing CreateTablet for tablet 97e1fe88ea8846eca38a1c2baa4e7b42 (DEFAULT_TABLE table=test-table [id=7124e1c7a93c4b3e988bdaa558f8707f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:46:18.157794 27573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97e1fe88ea8846eca38a1c2baa4e7b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:46:18.158908 27799 tablet_bootstrap.cc:492] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e: Bootstrap starting.
I20260807 22:46:18.159374 27800 tablet_bootstrap.cc:492] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465: Bootstrap starting.
I20260807 22:46:18.159798 27799 tablet_bootstrap.cc:654] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e: Neither blocks nor log segments found. Creating new log.
I20260807 22:46:18.160161 27800 tablet_bootstrap.cc:654] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465: Neither blocks nor log segments found. Creating new log.
I20260807 22:46:18.160635 27799 log.cc:824] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e: Log is configured to *not* fsync() on all Append() calls
I20260807 22:46:18.160847 27800 log.cc:824] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465: Log is configured to *not* fsync() on all Append() calls
I20260807 22:46:18.161163 27802 tablet_bootstrap.cc:492] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b: Bootstrap starting.
I20260807 22:46:18.161547 27799 tablet_bootstrap.cc:492] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e: No bootstrap required, opened a new log
I20260807 22:46:18.161582 27800 tablet_bootstrap.cc:492] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465: No bootstrap required, opened a new log
I20260807 22:46:18.161638 27799 ts_tablet_manager.cc:1397] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:46:18.161639 27800 ts_tablet_manager.cc:1397] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260807 22:46:18.161834 27802 tablet_bootstrap.cc:654] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b: Neither blocks nor log segments found. Creating new log.
I20260807 22:46:18.162333 27802 log.cc:824] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b: Log is configured to *not* fsync() on all Append() calls
I20260807 22:46:18.163020 27802 tablet_bootstrap.cc:492] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b: No bootstrap required, opened a new log
I20260807 22:46:18.163102 27802 ts_tablet_manager.cc:1397] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 22:46:18.163904 27800 raft_consensus.cc:348] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9191bf84a4c14e0fb72406bb797cc465" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34019 } } peers { permanent_uuid: "3e84edfe9ce641a580ccaab3b8b6970b" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40303 } } peers { permanent_uuid: "469c9a966f7b4fb088f161f47f7e316e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41895 } }
I20260807 22:46:18.163903 27799 raft_consensus.cc:348] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9191bf84a4c14e0fb72406bb797cc465" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34019 } } peers { permanent_uuid: "3e84edfe9ce641a580ccaab3b8b6970b" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40303 } } peers { permanent_uuid: "469c9a966f7b4fb088f161f47f7e316e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41895 } }
I20260807 22:46:18.164104 27800 raft_consensus.cc:374] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:46:18.164104 27799 raft_consensus.cc:374] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:46:18.164148 27800 raft_consensus.cc:729] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9191bf84a4c14e0fb72406bb797cc465, State: Initialized, Role: FOLLOWER
I20260807 22:46:18.164148 27799 raft_consensus.cc:729] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469c9a966f7b4fb088f161f47f7e316e, State: Initialized, Role: FOLLOWER
I20260807 22:46:18.164292 27800 consensus_queue.cc:260] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9191bf84a4c14e0fb72406bb797cc465" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34019 } } peers { permanent_uuid: "3e84edfe9ce641a580ccaab3b8b6970b" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40303 } } peers { permanent_uuid: "469c9a966f7b4fb088f161f47f7e316e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41895 } }
I20260807 22:46:18.164309 27799 consensus_queue.cc:260] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9191bf84a4c14e0fb72406bb797cc465" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34019 } } peers { permanent_uuid: "3e84edfe9ce641a580ccaab3b8b6970b" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40303 } } peers { permanent_uuid: "469c9a966f7b4fb088f161f47f7e316e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41895 } }
I20260807 22:46:18.164654 27800 ts_tablet_manager.cc:1428] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 22:46:18.164654 27799 ts_tablet_manager.cc:1428] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:46:18.164820 27782 heartbeater.cc:503] Master 127.16.98.190:44565 was elected leader, sending a full tablet report...
I20260807 22:46:18.164826 27504 heartbeater.cc:503] Master 127.16.98.190:44565 was elected leader, sending a full tablet report...
I20260807 22:46:18.165189 27802 raft_consensus.cc:348] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9191bf84a4c14e0fb72406bb797cc465" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34019 } } peers { permanent_uuid: "3e84edfe9ce641a580ccaab3b8b6970b" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40303 } } peers { permanent_uuid: "469c9a966f7b4fb088f161f47f7e316e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41895 } }
I20260807 22:46:18.165365 27802 raft_consensus.cc:374] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:46:18.165410 27802 raft_consensus.cc:729] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e84edfe9ce641a580ccaab3b8b6970b, State: Initialized, Role: FOLLOWER
I20260807 22:46:18.165524 27802 consensus_queue.cc:260] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9191bf84a4c14e0fb72406bb797cc465" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34019 } } peers { permanent_uuid: "3e84edfe9ce641a580ccaab3b8b6970b" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40303 } } peers { permanent_uuid: "469c9a966f7b4fb088f161f47f7e316e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41895 } }
I20260807 22:46:18.165831 27802 ts_tablet_manager.cc:1428] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:46:18.165905 27641 heartbeater.cc:503] Master 127.16.98.190:44565 was elected leader, sending a full tablet report...
W20260807 22:46:18.186594 27642 tablet.cc:2369] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:46:18.239919 27806 raft_consensus.cc:482] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:46:18.240108 27806 raft_consensus.cc:504] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9191bf84a4c14e0fb72406bb797cc465" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34019 } } peers { permanent_uuid: "3e84edfe9ce641a580ccaab3b8b6970b" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40303 } } peers { permanent_uuid: "469c9a966f7b4fb088f161f47f7e316e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41895 } }
I20260807 22:46:18.240547 27806 leader_election.cc:288] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9191bf84a4c14e0fb72406bb797cc465 (127.16.98.131:34019), 3e84edfe9ce641a580ccaab3b8b6970b (127.16.98.130:40303)
I20260807 22:46:18.244006 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:18.240783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:34019 (local address 127.16.98.129:42833)
0807 22:46:18.240945 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:18.240960 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:18.241046 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:18.241225 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:18.241226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:18.241228 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:46:18.241315 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:18.241317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.241996 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.241997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:18.242891 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:18.242894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.243685 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.243686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:18.243798 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:18.243803 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:46:18.243870 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 22:46:18.243919 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 22:46:18.244076 27795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:18.240849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:42833 (local address 127.16.98.131:34019)
0807 22:46:18.240926 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:18.240928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:18.241027 (+ 99us) server_negotiation.cc:407] Connection header received
0807 22:46:18.241133 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:18.241134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:18.241146 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:18.241207 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:46:18.241395 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.241901 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.242996 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.243610 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.243688 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:18.243699 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:18.243880 (+ 181us) server_negotiation.cc:299] Negotiation successful
0807 22:46:18.243994 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 22:46:18.244390 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:18.240924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:40303 (local address 127.16.98.129:36835)
0807 22:46:18.241096 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:18.241108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:46:18.241178 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:46:18.241315 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:46:18.241316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:46:18.241317 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:46:18.241404 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:18.241406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.242150 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.242151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:18.243055 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:46:18.243058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.244063 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.244064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:46:18.244174 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:18.244183 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 22:46:18.244265 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 22:46:18.244310 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":119,"threads_started":1}
I20260807 22:46:18.244411 27798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:18.241026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:36835 (local address 127.16.98.130:40303)
0807 22:46:18.241081 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:18.241084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:18.241165 (+ 81us) server_negotiation.cc:407] Connection header received
0807 22:46:18.241236 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:18.241237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:18.241248 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:18.241314 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:46:18.241482 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.242045 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.243210 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.243901 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.244059 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:18.244069 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:18.244294 (+ 225us) server_negotiation.cc:299] Negotiation successful
0807 22:46:18.244342 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 22:46:18.244441 27736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e1fe88ea8846eca38a1c2baa4e7b42" candidate_uuid: "469c9a966f7b4fb088f161f47f7e316e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9191bf84a4c14e0fb72406bb797cc465" is_pre_election: true
I20260807 22:46:18.244644 27736 raft_consensus.cc:2454] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469c9a966f7b4fb088f161f47f7e316e in term 0.
I20260807 22:46:18.244817 27593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e1fe88ea8846eca38a1c2baa4e7b42" candidate_uuid: "469c9a966f7b4fb088f161f47f7e316e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e84edfe9ce641a580ccaab3b8b6970b" is_pre_election: true
I20260807 22:46:18.244902 27391 leader_election.cc:302] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 469c9a966f7b4fb088f161f47f7e316e, 9191bf84a4c14e0fb72406bb797cc465; no voters:
I20260807 22:46:18.244993 27593 raft_consensus.cc:2454] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469c9a966f7b4fb088f161f47f7e316e in term 0.
I20260807 22:46:18.245057 27806 raft_consensus.cc:2781] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:46:18.245132 27806 raft_consensus.cc:482] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:46:18.245157 27806 raft_consensus.cc:3037] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:46:18.245961 27806 raft_consensus.cc:504] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9191bf84a4c14e0fb72406bb797cc465" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34019 } } peers { permanent_uuid: "3e84edfe9ce641a580ccaab3b8b6970b" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40303 } } peers { permanent_uuid: "469c9a966f7b4fb088f161f47f7e316e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41895 } }
I20260807 22:46:18.246073 27806 leader_election.cc:288] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [CANDIDATE]: Term 1 election: Requested vote from peers 9191bf84a4c14e0fb72406bb797cc465 (127.16.98.131:34019), 3e84edfe9ce641a580ccaab3b8b6970b (127.16.98.130:40303)
I20260807 22:46:18.246353 27593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e1fe88ea8846eca38a1c2baa4e7b42" candidate_uuid: "469c9a966f7b4fb088f161f47f7e316e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e84edfe9ce641a580ccaab3b8b6970b"
I20260807 22:46:18.246464 27593 raft_consensus.cc:3037] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:46:18.247331 27593 raft_consensus.cc:2454] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469c9a966f7b4fb088f161f47f7e316e in term 1.
I20260807 22:46:18.247470 27736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e1fe88ea8846eca38a1c2baa4e7b42" candidate_uuid: "469c9a966f7b4fb088f161f47f7e316e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9191bf84a4c14e0fb72406bb797cc465"
I20260807 22:46:18.247574 27391 leader_election.cc:302] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e84edfe9ce641a580ccaab3b8b6970b, 469c9a966f7b4fb088f161f47f7e316e; no voters:
I20260807 22:46:18.247581 27736 raft_consensus.cc:3037] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:46:18.247676 27806 raft_consensus.cc:2781] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:46:18.247843 27806 raft_consensus.cc:686] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [term 1 LEADER]: Becoming Leader. State: Replica: 469c9a966f7b4fb088f161f47f7e316e, State: Running, Role: LEADER
I20260807 22:46:18.247922 27806 consensus_queue.cc:237] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9191bf84a4c14e0fb72406bb797cc465" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34019 } } peers { permanent_uuid: "3e84edfe9ce641a580ccaab3b8b6970b" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40303 } } peers { permanent_uuid: "469c9a966f7b4fb088f161f47f7e316e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41895 } }
I20260807 22:46:18.248601 27736 raft_consensus.cc:2454] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469c9a966f7b4fb088f161f47f7e316e in term 1.
I20260807 22:46:18.248890 27303 catalog_manager.cc:5495] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e reported cstate change: term changed from 0 to 1, leader changed from <none> to 469c9a966f7b4fb088f161f47f7e316e (127.16.98.129). New cstate: current_term: 1 leader_uuid: "469c9a966f7b4fb088f161f47f7e316e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9191bf84a4c14e0fb72406bb797cc465" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e84edfe9ce641a580ccaab3b8b6970b" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469c9a966f7b4fb088f161f47f7e316e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 41895 } health_report { overall_health: HEALTHY } } }
W20260807 22:46:18.266367 27505 tablet.cc:2369] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:46:18.272984 27797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:18.270093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.16.98.129:41895)
0807 22:46:18.270169 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 22:46:18.270171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:46:18.270179 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:46:18.270219 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:46:18.270220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:46:18.270239 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:46:18.270431 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:46:18.270570 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.271153 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.271928 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:46:18.272159 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:46:18.272234 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:46:18.272420 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:46:18.272535 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:46:18.272767 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 22:46:18.272838 (+ 71us) server_negotiation.cc:299] Negotiation successful
0807 22:46:18.272861 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 22:46:18.275579 27593 raft_consensus.cc:1261] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b [term 1 FOLLOWER]: Refusing update from remote peer 469c9a966f7b4fb088f161f47f7e316e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:46:18.275607 27736 raft_consensus.cc:1261] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465 [term 1 FOLLOWER]: Refusing update from remote peer 469c9a966f7b4fb088f161f47f7e316e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:46:18.275986 27806 consensus_queue.cc:1035] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e84edfe9ce641a580ccaab3b8b6970b" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 40303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:46:18.276171 27809 consensus_queue.cc:1035] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 469c9a966f7b4fb088f161f47f7e316e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9191bf84a4c14e0fb72406bb797cc465" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:46:18.279016 27812 mvcc.cc:205] Tried to move back new op lower bound from 7316040819812909056 to 7316040819704266752. Current Snapshot: MvccSnapshot[applied={T|T < 7316040819812909056}]
I20260807 22:46:18.279407 27813 mvcc.cc:205] Tried to move back new op lower bound from 7316040819812909056 to 7316040819704266752. Current Snapshot: MvccSnapshot[applied={T|T < 7316040819812909056}]
I20260807 22:46:18.280330 27815 mvcc.cc:205] Tried to move back new op lower bound from 7316040819812909056 to 7316040819704266752. Current Snapshot: MvccSnapshot[applied={T|T < 7316040819812909056}]
I20260807 22:46:18.281883 27303 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:46:18.282027 27303 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:46:18.282956 27303 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb5a5b09f00943ae8bad9c7d6d2910a6: Sending DeleteTablet for 3 replicas of tablet 97e1fe88ea8846eca38a1c2baa4e7b42
I20260807 22:46:18.283351 27715 tablet_service.cc:1514] Processing DeleteTablet for tablet 97e1fe88ea8846eca38a1c2baa4e7b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:46:18 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:60790
I20260807 22:46:18.283389 27573 tablet_service.cc:1514] Processing DeleteTablet for tablet 97e1fe88ea8846eca38a1c2baa4e7b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:46:18 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:49344
I20260807 22:46:18.283605 27437 tablet_service.cc:1514] Processing DeleteTablet for tablet 97e1fe88ea8846eca38a1c2baa4e7b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:46:18 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:33750
I20260807 22:46:18.283737 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 27372
I20260807 22:46:18.283854 27824 tablet_replica.cc:330] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465: stopping tablet replica
I20260807 22:46:18.285004 27825 tablet_replica.cc:330] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b: stopping tablet replica
I20260807 22:46:18.285153 27825 raft_consensus.cc:2229] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:46:18.285308 27825 raft_consensus.cc:2258] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:46:18.285514 27824 raft_consensus.cc:2229] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:46:18.285661 27824 raft_consensus.cc:2258] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:46:18.286301 27824 ts_tablet_manager.cc:1905] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:46:18.286438 27825 ts_tablet_manager.cc:1905] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:46:18.288237 27825 ts_tablet_manager.cc:1918] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:46:18.288358 27825 log.cc:1196] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/97e1fe88ea8846eca38a1c2baa4e7b42
I20260807 22:46:18.288626 27824 ts_tablet_manager.cc:1918] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:46:18.288712 27824 log.cc:1196] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/97e1fe88ea8846eca38a1c2baa4e7b42
I20260807 22:46:18.288738 27825 ts_tablet_manager.cc:1939] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 3e84edfe9ce641a580ccaab3b8b6970b: Deleting consensus metadata
I20260807 22:46:18.289029 27824 ts_tablet_manager.cc:1939] T 97e1fe88ea8846eca38a1c2baa4e7b42 P 9191bf84a4c14e0fb72406bb797cc465: Deleting consensus metadata
I20260807 22:46:18.290014 27289 catalog_manager.cc:4841] TS 9191bf84a4c14e0fb72406bb797cc465 (127.16.98.131:34019): tablet 97e1fe88ea8846eca38a1c2baa4e7b42 (table test-table [id=7124e1c7a93c4b3e988bdaa558f8707f]) successfully deleted
I20260807 22:46:18.290096 27289 catalog_manager.cc:4841] TS 3e84edfe9ce641a580ccaab3b8b6970b (127.16.98.130:40303): tablet 97e1fe88ea8846eca38a1c2baa4e7b42 (table test-table [id=7124e1c7a93c4b3e988bdaa558f8707f]) successfully deleted
W20260807 22:46:18.294786 27290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:41895 (error 108)
I20260807 22:46:18.295188 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 27509
I20260807 22:46:18.296319 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:46:18.295403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:41895 (local address 127.0.0.1:33772)
0807 22:46:18.296211 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0807 22:46:18.296267 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:41895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":778}
W20260807 22:46:18.297605 27290 catalog_manager.cc:4584] TS 469c9a966f7b4fb088f161f47f7e316e (127.16.98.129:41895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97e1fe88ea8846eca38a1c2baa4e7b42: Network error: Client connection negotiation failed: client connection to 127.16.98.129:41895: connect: Connection refused (error 111)
I20260807 22:46:18.302116 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 27646
I20260807 22:46:18.309190 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 27270
2026-08-07T22:46:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2229 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:46:18 dist-test-slave-rgb1 krb5kdc[27830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:46:18 dist-test-slave-rgb1 krb5kdc[27830](info): set up 2 sockets
Aug 07 22:46:18 dist-test-slave-rgb1 krb5kdc[27830](info): commencing operation
krb5kdc: starting...
W20260807 22:46:19.433593 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:46:19 dist-test-slave-rgb1 krb5kdc[27830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:46:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.98.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.98.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/rangeradmin_127.16.98.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/rangeradmin_127.16.98.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.98.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.98.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/rangerlookup_127.16.98.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/rangerlookup_127.16.98.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.98.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.98.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/HTTP_127.16.98.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/HTTP_127.16.98.148@KRBTEST.COM.keytab.
I20260807 22:46:19.637128 16778 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskG4wJ7J/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/postgres -l logfile start
2026-08-07 22:46:22.846 UTC [27860] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 22:46:22.846 UTC [27860] LOG: listening on IPv4 address "127.16.98.148", port 34367
2026-08-07 22:46:22.852 UTC [27860] LOG: listening on Unix socket "/tmp/.s.PGSQL.34367"
2026-08-07 22:46:22.880 UTC [27863] LOG: database system was shut down at 2026-08-07 22:46:20 UTC
2026-08-07 22:46:22.886 UTC [27860] LOG: database system is ready to accept connections
I20260807 22:46:23.947304 16778 mini_postgres.cc:96] Postgres bound to 34367
2026-08-07 22:46:23.956 UTC [27872] FATAL: database "slave" does not exist
127.16.98.148:34367 - accepting connections
I20260807 22:46:23.957193 16778 mini_ranger.cc:161] Starting Ranger...
I20260807 22:46:23.977485 16778 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 22:46:24.632085 16778 mini_ranger.cc:87] Created ranger Postgres database
I20260807 22:46:24.632179 16778 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-admin
2026-08-07 22:46:24,971 [I] DB FLAVOR :POSTGRES
2026-08-07 22:46:24,972 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 22:46:24,972 [I] Checking connection..
2026-08-07 22:46:24,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 22:46:25,240 [I] Checking connection passed.
2026-08-07 22:46:25,240 [I] --------- Verifying version history table ---------
2026-08-07 22:46:25,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 22:46:25,538 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 22:46:25,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 22:46:25,794 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 22:46:25,794 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 22:46:25,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 22:46:26.057 UTC [27937] WARNING: there is no transaction in progress
2026-08-07 22:46:26,065 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 22:46:26,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 22:46:26,362 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 22:46:26,362 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 22:46:26,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 22:46:26,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 22:46:26,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rgb1.c.gcp-upstream.internal','N') ;"
2026-08-07 22:46:27,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 22:46:27,451 [I] Table x_portal_user does not exist in database ranger
2026-08-07 22:46:27,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 22:46:27,740 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 22:46:27,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 22:46:28,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 22:46:28,299 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 22:46:28,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 22:46:29.507 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:29.529 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:29.550 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:29.573 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:29.595 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:29.618 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:29.685 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:29.695 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:29.712 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:29.724 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:29.737 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:29.753 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:29.766 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:29.774 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:29.783 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:29.791 UTC [28047] WARNING: there is no transaction in progress
2026-08-07 22:46:30,731 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 22:46:30,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 22:46:31,004 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 22:46:31,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 22:46:31,281 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 22:46:31,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 22:46:31,550 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 22:46:31,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 22:46:31,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 22:46:32,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rgb1.c.gcp-upstream.internal';"
2026-08-07 22:46:32,412 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 22:46:32,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 22:46:32,690 [I] DB_PATCHES have already been applied
I20260807 22:46:32.697100 16778 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-admin:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskG4wJ7J/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 22:46:32.697185 16778 mini_ranger.cc:193] Using host: 127.16.98.148
I20260807 22:46:32.701606 16778 mini_ranger.cc:227] Ranger admin URL: http://127.16.98.148:36577
Aug 07, 2026 10:46:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 10:46:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 10:46:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 10:46:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 10:46:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 10:46:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 10:46:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 22:46:33.954257 16778 mini_ranger.cc:160] Time spent starting Ranger: real 9.997s user 0.003s sys 0.010s
Aug 07, 2026 10:46:58 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 10:46:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 07, 2026 10:46:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 10:46:58 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 10:46:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 10:46:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 10:46:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 10:46:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 10:46:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 10:46:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 10:46:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 10:46:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 10:46:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 10:46:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 10:46:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 10:46:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 10:46:58 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 10:46:59 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 22:46:59.959491 16778 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.98.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.98.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/rangerkms_127.16.98.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/rangerkms_127.16.98.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.98.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.98.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.98.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/HTTP_127.16.98.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.98.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/HTTP_127.16.98.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 22:47:00 dist-test-slave-rgb1 krb5kdc[27830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142820, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 22:47:00.095903 16778 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 22:47:00.111565 16778 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 22:47:01.370940 16778 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 22:47:01.371037 16778 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-kms
I20260807 22:47:01.371052 16778 mini_ranger_kms.cc:227] Using postgres at 127.16.98.148:34367
2026-08-07 22:47:02,217 [I] DB FLAVOR :POSTGRES
2026-08-07 22:47:02,217 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 22:47:02,217 [I] Checking connection
2026-08-07 22:47:02,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 22:47:02,515 [I] connection success
2026-08-07 22:47:02,516 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 22:47:02,516 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 22:47:02,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 22:47:02,789 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 22:47:02,789 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 22:47:02,790 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 22:47:02,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:34367/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 22:47:03,119 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 22:47:03.723887 16778 mini_ranger_kms.cc:310] Created kms service
I20260807 22:47:03.968604 16778 mini_ranger_kms.cc:326] Created kudu user
I20260807 22:47:04.030371 16778 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 22:47:04 dist-test-slave-rgb1 krb5kdc[27830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142820, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:47:04.510484 16778 mini_ranger_kms.cc:384] Added ranger policy
I20260807 22:47:04.510613 16778 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-kms:/tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 22:47:04.510634 16778 mini_ranger_kms.cc:240] Using host: 127.16.98.148
I20260807 22:47:04.515728 16778 mini_ranger_kms.cc:276] Ranger KMS PID: 28231
I20260807 22:47:04.515875 16778 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.98.148:48051
Aug 07, 2026 10:47:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 10:47:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 10:47:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 10:47:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 10:47:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 10:47:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 22:47:06 dist-test-slave-rgb1 krb5kdc[27830](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786142826, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.98.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:47:06.964461 16778 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.869s user 0.003s sys 0.332s
I20260807 22:47:06.964659 16778 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 22:47:06.964751 16778 mini_ranger_kms.cc:401] 127.16.98.148:48051/kms/v1/keys
Aug 07 22:47:07 dist-test-slave-rgb1 krb5kdc[27830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142826, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.98.148@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
Aug 07 22:47:07 dist-test-slave-rgb1 krb5kdc[27830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142826, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.98.148@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
Aug 07 22:47:10 dist-test-slave-rgb1 krb5kdc[27830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:47:10.309942 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:37249
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:40791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:37249
--ranger_config_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.98.148:48051/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:47:10.402968 28281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:47:10.403198 28281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:47:10.403221 28281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:47:10.404356 28281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:47:10.404397 28281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:47:10.404412 28281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:47:10.404423 28281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:47:10.404435 28281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:47:10.406153 28281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:40791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.98.148:48051/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:37249
--ranger_config_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:37249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:47:10.406370 28281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:47:10.406560 28281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:47:10.412930 28286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:47:10.412930 28287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:47:10.412997 28290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:47:10.412971 28281 server_base.cc:1034] running on GCE node
I20260807 22:47:10.413604 28281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:47:10.413949 28281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:47:10.415149 28281 hybrid_clock.cc:648] HybridClock initialized: now 1786142830415104 us; error 55 us; skew 500 ppm
Aug 07 22:47:10 dist-test-slave-rgb1 krb5kdc[27830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142830, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:47:10.425501 28281 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:47:10.426079 28281 webserver.cc:460] Webserver started at http://127.16.98.190:35799/ using document root <none> and password file <none>
I20260807 22:47:10.426333 28281 fs_manager.cc:359] Metadata directory not provided
I20260807 22:47:10.426378 28281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:47:10.426510 28281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:47:10 dist-test-slave-rgb1 krb5kdc[27830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142830, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:47:10.481869 28281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "6e371c3dc1e2464fb3a43be666094d08"
format_stamp: "Formatted at 2026-08-07 22:47:10 on dist-test-slave-rgb1"
server_key: "944f87ed05ef30fe43fb2c0e96c99ee6"
server_key_iv: "56360c59cd49e5197bc01423898b72f1"
server_key_version: "kuduclusterkey@0"
I20260807 22:47:10.482302 28281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "6e371c3dc1e2464fb3a43be666094d08"
format_stamp: "Formatted at 2026-08-07 22:47:10 on dist-test-slave-rgb1"
server_key: "944f87ed05ef30fe43fb2c0e96c99ee6"
server_key_iv: "56360c59cd49e5197bc01423898b72f1"
server_key_version: "kuduclusterkey@0"
I20260807 22:47:10.484133 28281 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260807 22:47:10.501550 28297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:47:10.501847 28281 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:47:10.501971 28281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "6e371c3dc1e2464fb3a43be666094d08"
format_stamp: "Formatted at 2026-08-07 22:47:10 on dist-test-slave-rgb1"
server_key: "944f87ed05ef30fe43fb2c0e96c99ee6"
server_key_iv: "56360c59cd49e5197bc01423898b72f1"
server_key_version: "kuduclusterkey@0"
I20260807 22:47:10.502084 28281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:47:10.518908 28281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:47:10.521240 28281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:47:10.521425 28281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:47:10.527177 28281 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:37249
I20260807 22:47:10.527230 28349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:37249 every 8 connection(s)
I20260807 22:47:10.527637 28281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:47:10.528443 28350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:47:10.529031 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 28281
I20260807 22:47:10.529150 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:47:10.529407 16778 external_mini_cluster.cc:1428] Setting key be65adc72fc51ad469d10624bce3b4cc
I20260807 22:47:10.532137 28350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08: Bootstrap starting.
I20260807 22:47:10.532889 28350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08: Neither blocks nor log segments found. Creating new log.
I20260807 22:47:10.533215 28350 log.cc:824] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08: Log is configured to *not* fsync() on all Append() calls
I20260807 22:47:10.534065 28350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08: No bootstrap required, opened a new log
I20260807 22:47:10.536716 28350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e371c3dc1e2464fb3a43be666094d08" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37249 } }
I20260807 22:47:10.536907 28350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:47:10.536947 28350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e371c3dc1e2464fb3a43be666094d08, State: Initialized, Role: FOLLOWER
Aug 07 22:47:10 dist-test-slave-rgb1 krb5kdc[27830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:47:10.537166 28350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e371c3dc1e2464fb3a43be666094d08" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37249 } }
I20260807 22:47:10.537257 28350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:47:10.537317 28350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:47:10.537367 28350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:47:10.538142 28350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e371c3dc1e2464fb3a43be666094d08" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37249 } }
I20260807 22:47:10.538261 28350 leader_election.cc:302] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e371c3dc1e2464fb3a43be666094d08; no voters:
I20260807 22:47:10.538463 28350 leader_election.cc:288] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:47:10.538532 28356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:47:10.538769 28356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [term 1 LEADER]: Becoming Leader. State: Replica: 6e371c3dc1e2464fb3a43be666094d08, State: Running, Role: LEADER
I20260807 22:47:10.538829 28350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:47:10.538964 28356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e371c3dc1e2464fb3a43be666094d08" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37249 } }
I20260807 22:47:10.539496 28356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e371c3dc1e2464fb3a43be666094d08. Latest consensus state: current_term: 1 leader_uuid: "6e371c3dc1e2464fb3a43be666094d08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e371c3dc1e2464fb3a43be666094d08" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37249 } } }
I20260807 22:47:10.539654 28356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [sys.catalog]: This master's current role is: LEADER
I20260807 22:47:10.540692 28350 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 22:47:10.541471 28357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e371c3dc1e2464fb3a43be666094d08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e371c3dc1e2464fb3a43be666094d08" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37249 } } }
I20260807 22:47:10.541527 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:10.530114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50404 (local address 127.16.98.190:37249)
0807 22:47:10.530306 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 22:47:10.530316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:47:10.530349 (+ 33us) server_negotiation.cc:407] Connection header received
0807 22:47:10.530910 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:47:10.530919 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:47:10.530983 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:47:10.531138 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:47:10.532138 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:10.533564 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:10.534450 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:10.534925 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:10.535053 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:10.537633 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:47:10.537637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:47:10.537637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:47:10.537650 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:47:10.539787 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:47:10.540235 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:47:10.540238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:47:10.540238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:47:10.540330 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:47:10.540575 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:47:10.540576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:47:10.540576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:47:10.540894 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:47:10.541090 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 22:47:10.541167 (+ 77us) server_negotiation.cc:299] Negotiation successful
0807 22:47:10.541288 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
I20260807 22:47:10.541539 28357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08 [sys.catalog]: This master's current role is: LEADER
I20260807 22:47:10.545320 28365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:47:10.546204 28365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:47:10.549094 28365 catalog_manager.cc:1347] Generated new cluster ID: 503a2ded784a4adbb377202b887d5bb0
I20260807 22:47:10.549155 28365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:47:10.560691 28365 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:47:10.561491 28365 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:47:10.567011 28365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08: Generated new TSK 0
I20260807 22:47:10.567209 28365 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:47:11.357965 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.98.148:48051/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37249
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:40791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:47:11.467077 28396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:47:11.467362 28396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:47:11.467413 28396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:47:11.468940 28396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:47:11.468997 28396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:47:11.469029 28396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:47:11.470842 28396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:40791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.98.148:48051/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:47:11.471140 28396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:47:11.471449 28396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:47:11.478852 28402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:47:11.478852 28404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:47:11.478852 28401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:47:11.481335 28396 server_base.cc:1034] running on GCE node
I20260807 22:47:11.481609 28396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:47:11.482087 28396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:47:11.483351 28396 hybrid_clock.cc:648] HybridClock initialized: now 1786142831483329 us; error 39 us; skew 500 ppm
Aug 07 22:47:11 dist-test-slave-rgb1 krb5kdc[27830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142831, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:47:11.487166 28396 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:47:11.488160 28396 webserver.cc:460] Webserver started at http://127.16.98.129:39903/ using document root <none> and password file <none>
I20260807 22:47:11.488472 28396 fs_manager.cc:359] Metadata directory not provided
I20260807 22:47:11.488597 28396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:47:11.488777 28396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:47:11 dist-test-slave-rgb1 krb5kdc[27830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142831, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:47:11.504766 28396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "33ff12597aad496694007454d68e99a6"
format_stamp: "Formatted at 2026-08-07 22:47:11 on dist-test-slave-rgb1"
server_key: "2357065ec509b9ccdc9699b57d65525f"
server_key_iv: "31b8223152f24991ec9da22c686f1b6c"
server_key_version: "kuduclusterkey@0"
I20260807 22:47:11.505337 28396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "33ff12597aad496694007454d68e99a6"
format_stamp: "Formatted at 2026-08-07 22:47:11 on dist-test-slave-rgb1"
server_key: "2357065ec509b9ccdc9699b57d65525f"
server_key_iv: "31b8223152f24991ec9da22c686f1b6c"
server_key_version: "kuduclusterkey@0"
I20260807 22:47:11.507893 28396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260807 22:47:11.523983 28412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:47:11.524209 28396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260807 22:47:11.524312 28396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "33ff12597aad496694007454d68e99a6"
format_stamp: "Formatted at 2026-08-07 22:47:11 on dist-test-slave-rgb1"
server_key: "2357065ec509b9ccdc9699b57d65525f"
server_key_iv: "31b8223152f24991ec9da22c686f1b6c"
server_key_version: "kuduclusterkey@0"
I20260807 22:47:11.524447 28396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:47:11.544845 28396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:47:11.547499 28396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:47:11.547710 28396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:47:11.548000 28396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:47:11.548429 28396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:47:11.548485 28396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:47:11.548525 28396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:47:11.548552 28396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:47:11.556748 28396 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:35831
I20260807 22:47:11.557231 28396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:47:11.559024 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 28396
I20260807 22:47:11.559129 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:47:11.559334 16778 external_mini_cluster.cc:1428] Setting key 097d2c74ef2393e6f6bcb39f574f7875
I20260807 22:47:11.565507 28526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:35831 every 8 connection(s)
Aug 07 22:47:11 dist-test-slave-rgb1 krb5kdc[27830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142831, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
I20260807 22:47:11.588810 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:11.566417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:40979 (local address 127.16.98.190:37249)
0807 22:47:11.567569 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0807 22:47:11.567572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:47:11.567580 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:47:11.567606 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:47:11.567607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:47:11.567620 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:47:11.567684 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:47:11.568660 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:11.569562 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:11.571244 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:11.571586 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:11.571676 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:11.583974 (+ 12298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:47:11.583977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:47:11.583977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:47:11.583988 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:47:11.586127 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:47:11.587094 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:47:11.587095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:47:11.587096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:47:11.587166 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:47:11.587508 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:47:11.587509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:47:11.587509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:47:11.587669 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:47:11.588014 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0807 22:47:11.588589 (+ 575us) server_negotiation.cc:299] Negotiation successful
0807 22:47:11.588695 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
I20260807 22:47:11.592280 28530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:11.566364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:37249 (local address 127.16.98.129:40979)
0807 22:47:11.566674 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0807 22:47:11.566719 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 22:47:11.567349 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:47:11.567821 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:47:11.567826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:47:11.568137 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:47:11.568479 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:11.568486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:11.569737 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:11.569738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:11.571132 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:11.571136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:11.571714 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:11.571716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:11.571879 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:11.572646 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:47:11.572657 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:47:11.583772 (+ 11115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:47:11.586537 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:47:11.586549 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:47:11.586553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:47:11.586988 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:47:11.587242 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:47:11.587242 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:47:11.587243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:47:11.587412 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:47:11.588105 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:47:11.588112 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:47:11.588309 (+ 197us) client_negotiation.cc:769] Sending connection context
0807 22:47:11.591752 (+ 3443us) client_negotiation.cc:240] Negotiation successful
0807 22:47:11.591985 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":125,"threads_started":1}
I20260807 22:47:11.593336 28527 heartbeater.cc:347] Connected to a master server at 127.16.98.190:37249
I20260807 22:47:11.593448 28527 heartbeater.cc:464] Registering TS with master...
I20260807 22:47:11.593663 28527 heartbeater.cc:511] Master 127.16.98.190:37249 requested a full tablet report, sending...
I20260807 22:47:11.594336 28314 ts_manager.cc:194] Registered new tserver with Master: 33ff12597aad496694007454d68e99a6 (127.16.98.129:35831)
I20260807 22:47:11.595242 28314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:40979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:47:11.625849 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.98.148:48051/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37249
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:40791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:47:11 dist-test-slave-rgb1 krb5kdc[27830](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786142831, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 22:47:11.769415 28533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:47:11.769742 28533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:47:11.769796 28533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:47:11.771667 28533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:47:11.771742 28533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:47:11.771795 28533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:47:11.775709 28533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:40791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.98.148:48051/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:47:11.776089 28533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:47:11.776440 28533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 22:47:11.786693 28533 server_base.cc:1034] running on GCE node
W20260807 22:47:11.786931 28543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:47:11.787191 28542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:47:11.787474 28545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:47:11.787735 28533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:47:11.788060 28533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:47:11.789232 28533 hybrid_clock.cc:648] HybridClock initialized: now 1786142831789208 us; error 40 us; skew 500 ppm
Aug 07 22:47:11 dist-test-slave-rgb1 krb5kdc[27830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142831, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:47:11.793166 28533 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:47:11.793777 28533 webserver.cc:460] Webserver started at http://127.16.98.130:33833/ using document root <none> and password file <none>
I20260807 22:47:11.794154 28533 fs_manager.cc:359] Metadata directory not provided
I20260807 22:47:11.794281 28533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:47:11.794471 28533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:47:11 dist-test-slave-rgb1 krb5kdc[27830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142831, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:47:11.814498 28533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "dfd5114296a5416b8af8ca70b1e8f985"
format_stamp: "Formatted at 2026-08-07 22:47:11 on dist-test-slave-rgb1"
server_key: "4318b11b11bfa852ce78d03312bbaac9"
server_key_iv: "f6b3c92c081bbd43a29d415373cc881e"
server_key_version: "kuduclusterkey@0"
I20260807 22:47:11.815060 28533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "dfd5114296a5416b8af8ca70b1e8f985"
format_stamp: "Formatted at 2026-08-07 22:47:11 on dist-test-slave-rgb1"
server_key: "4318b11b11bfa852ce78d03312bbaac9"
server_key_iv: "f6b3c92c081bbd43a29d415373cc881e"
server_key_version: "kuduclusterkey@0"
I20260807 22:47:11.819085 28533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260807 22:47:11.836477 28552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:47:11.836782 28533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:47:11.836877 28533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "dfd5114296a5416b8af8ca70b1e8f985"
format_stamp: "Formatted at 2026-08-07 22:47:11 on dist-test-slave-rgb1"
server_key: "4318b11b11bfa852ce78d03312bbaac9"
server_key_iv: "f6b3c92c081bbd43a29d415373cc881e"
server_key_version: "kuduclusterkey@0"
I20260807 22:47:11.836974 28533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:47:11.852012 28533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:47:11.854701 28533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:47:11.854880 28533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:47:11.855178 28533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:47:11.855604 28533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:47:11.855661 28533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:47:11.855707 28533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:47:11.855736 28533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:47:11.863701 28533 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:38071
I20260807 22:47:11.864297 28533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:47:11.865345 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 28533
I20260807 22:47:11.865471 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:47:11.865666 16778 external_mini_cluster.cc:1428] Setting key 69329b313b958278e452fa19389180e3
I20260807 22:47:11.870056 28667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:38071 every 8 connection(s)
Aug 07 22:47:11 dist-test-slave-rgb1 krb5kdc[27830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142831, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:47:11.885136 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:11.865600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:48429 (local address 127.16.98.190:37249)
0807 22:47:11.865661 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 22:47:11.865664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:47:11.865918 (+ 254us) server_negotiation.cc:407] Connection header received
0807 22:47:11.866757 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:47:11.866759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:47:11.866780 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:47:11.866814 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:47:11.867791 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:11.868717 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:11.870596 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:11.870974 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:11.871025 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:11.879069 (+ 8044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:47:11.879071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:47:11.879072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:47:11.879082 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:47:11.881169 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:47:11.883662 (+ 2493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:47:11.883665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:47:11.883665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:47:11.883740 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:47:11.884120 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:47:11.884121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:47:11.884121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:47:11.884291 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:47:11.884340 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0807 22:47:11.884885 (+ 545us) server_negotiation.cc:299] Negotiation successful
0807 22:47:11.885020 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 22:47:11.886096 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:11.865254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:37249 (local address 127.16.98.130:48429)
0807 22:47:11.865760 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0807 22:47:11.865820 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 22:47:11.866616 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:47:11.867032 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:47:11.867036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:47:11.867342 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:47:11.867667 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:11.867674 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:11.868863 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:11.868864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:11.870477 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:11.870480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:11.871140 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:11.871141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:11.871307 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:11.876852 (+ 5545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:47:11.876866 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:47:11.878895 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:47:11.881315 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:47:11.881326 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:47:11.881330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:47:11.881711 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:47:11.883834 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:47:11.883835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:47:11.883835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:47:11.884026 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:47:11.884410 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:47:11.884418 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:47:11.884657 (+ 239us) client_negotiation.cc:769] Sending connection context
0807 22:47:11.885589 (+ 932us) client_negotiation.cc:240] Negotiation successful
0807 22:47:11.885818 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":98,"threads_started":1}
I20260807 22:47:11.886930 28669 heartbeater.cc:347] Connected to a master server at 127.16.98.190:37249
I20260807 22:47:11.887053 28669 heartbeater.cc:464] Registering TS with master...
I20260807 22:47:11.887293 28669 heartbeater.cc:511] Master 127.16.98.190:37249 requested a full tablet report, sending...
I20260807 22:47:11.887727 28314 ts_manager.cc:194] Registered new tserver with Master: dfd5114296a5416b8af8ca70b1e8f985 (127.16.98.130:38071)
I20260807 22:47:11.888346 28314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:48429
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:47:11.928337 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.98.148:48051/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37249
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:40791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:47:12.063647 28674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:47:12.064034 28674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:47:12.064124 28674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:47:12.065966 28674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:47:12.066202 28674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:47:12.066291 28674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:47:12.068522 28674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:40791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.98.148:48051/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:47:12.069070 28674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:47:12.069407 28674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:47:12.076332 28680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:47:12.076691 28682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:47:12.076826 28679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:47:12.078056 28674 server_base.cc:1034] running on GCE node
I20260807 22:47:12.078274 28674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:47:12.078545 28674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:47:12.079711 28674 hybrid_clock.cc:648] HybridClock initialized: now 1786142832079690 us; error 38 us; skew 500 ppm
Aug 07 22:47:12 dist-test-slave-rgb1 krb5kdc[27830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:47:12.083825 28674 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:47:12.084662 28674 webserver.cc:460] Webserver started at http://127.16.98.131:34817/ using document root <none> and password file <none>
I20260807 22:47:12.085002 28674 fs_manager.cc:359] Metadata directory not provided
I20260807 22:47:12.085067 28674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:47:12.085386 28674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:47:12 dist-test-slave-rgb1 krb5kdc[27830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:47:12.108100 28674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "a9058e666bc1475dbecf38e418326a78"
format_stamp: "Formatted at 2026-08-07 22:47:12 on dist-test-slave-rgb1"
server_key: "51f718d35f9134af9a4c095e2a1c0a23"
server_key_iv: "47d92abe41436a0676497affb858ebe1"
server_key_version: "kuduclusterkey@0"
I20260807 22:47:12.108654 28674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "a9058e666bc1475dbecf38e418326a78"
format_stamp: "Formatted at 2026-08-07 22:47:12 on dist-test-slave-rgb1"
server_key: "51f718d35f9134af9a4c095e2a1c0a23"
server_key_iv: "47d92abe41436a0676497affb858ebe1"
server_key_version: "kuduclusterkey@0"
I20260807 22:47:12.110862 28674 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260807 22:47:12.132529 28689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:47:12.132843 28674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260807 22:47:12.132969 28674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "a9058e666bc1475dbecf38e418326a78"
format_stamp: "Formatted at 2026-08-07 22:47:12 on dist-test-slave-rgb1"
server_key: "51f718d35f9134af9a4c095e2a1c0a23"
server_key_iv: "47d92abe41436a0676497affb858ebe1"
server_key_version: "kuduclusterkey@0"
I20260807 22:47:12.133091 28674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:47:12.151609 28674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:47:12.154561 28674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:47:12.154776 28674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:47:12.155061 28674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:47:12.155516 28674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:47:12.155588 28674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:47:12.155635 28674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:47:12.155665 28674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:47:12.165820 28674 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:41469
I20260807 22:47:12.165874 28803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:41469 every 8 connection(s)
I20260807 22:47:12.166312 28674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:47:12.171658 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 28674
I20260807 22:47:12.171788 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:47:12.172045 16778 external_mini_cluster.cc:1428] Setting key 7bdd32f975bb1e85b066237400362009
Aug 07 22:47:12 dist-test-slave-rgb1 krb5kdc[27830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:47:12.180240 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.167442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:34867 (local address 127.16.98.190:37249)
0807 22:47:12.167994 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0807 22:47:12.167997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:47:12.168047 (+ 50us) server_negotiation.cc:407] Connection header received
0807 22:47:12.168666 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:47:12.168668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:47:12.168680 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:47:12.168734 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:47:12.169338 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.169854 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.172287 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.172636 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.172684 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.175493 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:47:12.175495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:47:12.175496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:47:12.175506 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:47:12.177700 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:47:12.178730 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:47:12.178733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:47:12.178734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:47:12.178948 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:47:12.179423 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:47:12.179424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:47:12.179425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:47:12.179581 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:47:12.179671 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 22:47:12.180018 (+ 347us) server_negotiation.cc:299] Negotiation successful
0807 22:47:12.180133 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523}
I20260807 22:47:12.180485 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.167444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:37249 (local address 127.16.98.131:34867)
0807 22:47:12.167903 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0807 22:47:12.167950 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 22:47:12.168496 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:47:12.168813 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:47:12.168816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:47:12.169034 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:47:12.169238 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.169242 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.169955 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.169957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.172156 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.172161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.172788 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.172789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.172942 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.173832 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:47:12.173842 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:47:12.175360 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:47:12.178079 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:47:12.178095 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:47:12.178098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:47:12.178497 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:47:12.179101 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:47:12.179104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:47:12.179104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:47:12.179290 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:47:12.179674 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:47:12.179683 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:47:12.179857 (+ 174us) client_negotiation.cc:769] Sending connection context
0807 22:47:12.179992 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 22:47:12.180189 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":363,"threads_started":1}
I20260807 22:47:12.181296 28804 heartbeater.cc:347] Connected to a master server at 127.16.98.190:37249
I20260807 22:47:12.181433 28804 heartbeater.cc:464] Registering TS with master...
I20260807 22:47:12.181649 28804 heartbeater.cc:511] Master 127.16.98.190:37249 requested a full tablet report, sending...
I20260807 22:47:12.182132 28314 ts_manager.cc:194] Registered new tserver with Master: a9058e666bc1475dbecf38e418326a78 (127.16.98.131:41469)
I20260807 22:47:12.182569 28314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:34867
I20260807 22:47:12.183981 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:47:12.194207 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.185810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.16.98.190:37249)
0807 22:47:12.185871 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 22:47:12.185874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:47:12.186069 (+ 195us) server_negotiation.cc:407] Connection header received
0807 22:47:12.186164 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:47:12.186165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:47:12.186178 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:47:12.186212 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:47:12.186921 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.187465 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.188465 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.188816 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.188894 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.190697 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:47:12.190699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:47:12.190699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:47:12.190708 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:47:12.192725 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:47:12.193192 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:47:12.193193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:47:12.193194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:47:12.193256 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:47:12.193507 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:47:12.193509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:47:12.193509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:47:12.193666 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:47:12.193749 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 22:47:12.194001 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 22:47:12.194094 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 22:47:12.197099 28314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:47:12.200273 28314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:47:12.213728 28819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.205382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:35831 (local address 127.0.0.1:42652)
0807 22:47:12.208713 (+ 3331us) negotiation.cc:107] Waiting for socket to connect
0807 22:47:12.208722 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:47:12.208771 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:47:12.209048 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:47:12.209051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:47:12.209053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:47:12.209129 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.209130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.210438 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.210439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.211714 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.211718 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.213327 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.213328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.213443 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.213459 (+ 16us) client_negotiation.cc:769] Sending connection context
0807 22:47:12.213532 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 22:47:12.213599 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3315,"thread_start_us":78,"threads_started":1}
I20260807 22:47:12.218104 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.205698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34912 (local address 127.16.98.130:38071)
0807 22:47:12.212111 (+ 6413us) server_negotiation.cc:206] Beginning negotiation
0807 22:47:12.212117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:47:12.212129 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:47:12.212170 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:47:12.212172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:47:12.212189 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:47:12.212266 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:47:12.212474 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.213515 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.215183 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.216086 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.216177 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.216266 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 22:47:12.217926 (+ 1660us) server_negotiation.cc:299] Negotiation successful
0807 22:47:12.217980 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6364,"thread_start_us":100,"threads_started":1}
I20260807 22:47:12.218181 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.205550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:38071 (local address 127.0.0.1:34912)
0807 22:47:12.211861 (+ 6311us) negotiation.cc:107] Waiting for socket to connect
0807 22:47:12.211871 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:47:12.211914 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:47:12.212355 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:47:12.212355 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:47:12.212358 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:47:12.212401 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.212402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.213621 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.213622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.215044 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.215047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.216439 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.216440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.216500 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.216504 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:47:12.216550 (+ 46us) client_negotiation.cc:240] Negotiation successful
0807 22:47:12.218105 (+ 1555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6290,"mutex_wait_us":1480,"thread_start_us":69,"threads_started":1}
I20260807 22:47:12.220496 28601 tablet_service.cc:1467] Processing CreateTablet for tablet b9ca6375b0b7483c9ec4f735a03c9ae0 (DEFAULT_TABLE table=test-table [id=efe6836955944a0e96b085cb0f802457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:47:12.221019 28601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ca6375b0b7483c9ec4f735a03c9ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:47:12.225306 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.205763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.16.98.129:35831)
0807 22:47:12.208656 (+ 2893us) server_negotiation.cc:206] Beginning negotiation
0807 22:47:12.208662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:47:12.208822 (+ 160us) server_negotiation.cc:407] Connection header received
0807 22:47:12.208859 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:47:12.208861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:47:12.208879 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:47:12.208955 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:47:12.209205 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.210331 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.211818 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.212795 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.212888 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.225105 (+ 12217us) server_negotiation.cc:1035] Waiting for connection context
0807 22:47:12.225145 (+ 40us) server_negotiation.cc:299] Negotiation successful
0807 22:47:12.225215 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2855,"thread_start_us":55,"threads_started":1}
I20260807 22:47:12.226928 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.205111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:41469 (local address 127.0.0.1:42464)
0807 22:47:12.208318 (+ 3207us) negotiation.cc:107] Waiting for socket to connect
0807 22:47:12.208377 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 22:47:12.208463 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:47:12.215177 (+ 6714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:47:12.215178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:47:12.215180 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:47:12.215214 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.215215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.216426 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.216427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.217768 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.217772 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.226586 (+ 8814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.226588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.226681 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.226687 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 22:47:12.226764 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 22:47:12.226822 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3166,"thread_start_us":106,"threads_started":1}
I20260807 22:47:12.227351 28460 tablet_service.cc:1467] Processing CreateTablet for tablet b9ca6375b0b7483c9ec4f735a03c9ae0 (DEFAULT_TABLE table=test-table [id=efe6836955944a0e96b085cb0f802457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:47:12.227736 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ca6375b0b7483c9ec4f735a03c9ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:47:12.231675 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.205280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42464 (local address 127.16.98.131:41469)
0807 22:47:12.213858 (+ 8578us) server_negotiation.cc:206] Beginning negotiation
0807 22:47:12.213863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:47:12.213874 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:47:12.213913 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:47:12.213916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:47:12.214005 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:47:12.215282 (+ 1277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:47:12.215361 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.216323 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.225685 (+ 9362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.226488 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.226580 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.226658 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 22:47:12.231529 (+ 4871us) server_negotiation.cc:299] Negotiation successful
0807 22:47:12.231589 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8533,"thread_start_us":99,"threads_started":1}
I20260807 22:47:12.232270 28823 tablet_bootstrap.cc:492] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985: Bootstrap starting.
I20260807 22:47:12.232547 28736 tablet_service.cc:1467] Processing CreateTablet for tablet b9ca6375b0b7483c9ec4f735a03c9ae0 (DEFAULT_TABLE table=test-table [id=efe6836955944a0e96b085cb0f802457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:47:12.232910 28824 tablet_bootstrap.cc:492] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6: Bootstrap starting.
I20260807 22:47:12.232946 28736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9ca6375b0b7483c9ec4f735a03c9ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:47:12.233752 28823 tablet_bootstrap.cc:654] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985: Neither blocks nor log segments found. Creating new log.
I20260807 22:47:12.234490 28824 tablet_bootstrap.cc:654] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6: Neither blocks nor log segments found. Creating new log.
I20260807 22:47:12.234939 28823 log.cc:824] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985: Log is configured to *not* fsync() on all Append() calls
I20260807 22:47:12.235357 28824 log.cc:824] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6: Log is configured to *not* fsync() on all Append() calls
I20260807 22:47:12.236706 28824 tablet_bootstrap.cc:492] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6: No bootstrap required, opened a new log
I20260807 22:47:12.236864 28824 ts_tablet_manager.cc:1397] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 22:47:12.239631 28824 raft_consensus.cc:348] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff12597aad496694007454d68e99a6" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 35831 } } peers { permanent_uuid: "dfd5114296a5416b8af8ca70b1e8f985" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38071 } } peers { permanent_uuid: "a9058e666bc1475dbecf38e418326a78" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41469 } }
I20260807 22:47:12.239967 28824 raft_consensus.cc:374] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:47:12.240058 28824 raft_consensus.cc:729] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ff12597aad496694007454d68e99a6, State: Initialized, Role: FOLLOWER
I20260807 22:47:12.240293 28824 consensus_queue.cc:260] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff12597aad496694007454d68e99a6" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 35831 } } peers { permanent_uuid: "dfd5114296a5416b8af8ca70b1e8f985" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38071 } } peers { permanent_uuid: "a9058e666bc1475dbecf38e418326a78" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41469 } }
I20260807 22:47:12.240674 28824 ts_tablet_manager.cc:1428] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260807 22:47:12.240837 28527 heartbeater.cc:503] Master 127.16.98.190:37249 was elected leader, sending a full tablet report...
I20260807 22:47:12.242383 28823 tablet_bootstrap.cc:492] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985: No bootstrap required, opened a new log
I20260807 22:47:12.242513 28823 ts_tablet_manager.cc:1397] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260807 22:47:12.244864 28823 raft_consensus.cc:348] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff12597aad496694007454d68e99a6" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 35831 } } peers { permanent_uuid: "dfd5114296a5416b8af8ca70b1e8f985" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38071 } } peers { permanent_uuid: "a9058e666bc1475dbecf38e418326a78" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41469 } }
I20260807 22:47:12.245139 28823 raft_consensus.cc:374] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:47:12.245220 28823 raft_consensus.cc:729] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd5114296a5416b8af8ca70b1e8f985, State: Initialized, Role: FOLLOWER
I20260807 22:47:12.245401 28823 consensus_queue.cc:260] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff12597aad496694007454d68e99a6" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 35831 } } peers { permanent_uuid: "dfd5114296a5416b8af8ca70b1e8f985" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38071 } } peers { permanent_uuid: "a9058e666bc1475dbecf38e418326a78" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41469 } }
I20260807 22:47:12.245752 28823 ts_tablet_manager.cc:1428] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 22:47:12.245909 28669 heartbeater.cc:503] Master 127.16.98.190:37249 was elected leader, sending a full tablet report...
I20260807 22:47:12.247242 28827 tablet_bootstrap.cc:492] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78: Bootstrap starting.
I20260807 22:47:12.248428 28827 tablet_bootstrap.cc:654] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78: Neither blocks nor log segments found. Creating new log.
I20260807 22:47:12.249387 28827 log.cc:824] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78: Log is configured to *not* fsync() on all Append() calls
I20260807 22:47:12.250278 28827 tablet_bootstrap.cc:492] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78: No bootstrap required, opened a new log
I20260807 22:47:12.250346 28827 ts_tablet_manager.cc:1397] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:47:12.252297 28827 raft_consensus.cc:348] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff12597aad496694007454d68e99a6" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 35831 } } peers { permanent_uuid: "dfd5114296a5416b8af8ca70b1e8f985" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38071 } } peers { permanent_uuid: "a9058e666bc1475dbecf38e418326a78" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41469 } }
I20260807 22:47:12.252463 28827 raft_consensus.cc:374] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:47:12.252492 28827 raft_consensus.cc:729] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9058e666bc1475dbecf38e418326a78, State: Initialized, Role: FOLLOWER
I20260807 22:47:12.252632 28827 consensus_queue.cc:260] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff12597aad496694007454d68e99a6" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 35831 } } peers { permanent_uuid: "dfd5114296a5416b8af8ca70b1e8f985" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38071 } } peers { permanent_uuid: "a9058e666bc1475dbecf38e418326a78" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41469 } }
I20260807 22:47:12.252892 28827 ts_tablet_manager.cc:1428] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:47:12.253526 28804 heartbeater.cc:503] Master 127.16.98.190:37249 was elected leader, sending a full tablet report...
I20260807 22:47:12.269356 28829 raft_consensus.cc:482] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:47:12.269510 28829 raft_consensus.cc:504] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff12597aad496694007454d68e99a6" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 35831 } } peers { permanent_uuid: "dfd5114296a5416b8af8ca70b1e8f985" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38071 } } peers { permanent_uuid: "a9058e666bc1475dbecf38e418326a78" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41469 } }
I20260807 22:47:12.269894 28829 leader_election.cc:288] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33ff12597aad496694007454d68e99a6 (127.16.98.129:35831), a9058e666bc1475dbecf38e418326a78 (127.16.98.131:41469)
I20260807 22:47:12.273537 28828 raft_consensus.cc:482] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:47:12.273659 28828 raft_consensus.cc:504] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff12597aad496694007454d68e99a6" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 35831 } } peers { permanent_uuid: "dfd5114296a5416b8af8ca70b1e8f985" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38071 } } peers { permanent_uuid: "a9058e666bc1475dbecf38e418326a78" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41469 } }
I20260807 22:47:12.274036 28828 leader_election.cc:288] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfd5114296a5416b8af8ca70b1e8f985 (127.16.98.130:38071), a9058e666bc1475dbecf38e418326a78 (127.16.98.131:41469)
I20260807 22:47:12.276458 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.270189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:41469 (local address 127.16.98.130:54053)
0807 22:47:12.270253 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0807 22:47:12.270272 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 22:47:12.270352 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:47:12.271045 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:47:12.271046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:47:12.271048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:47:12.271122 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.271124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.272049 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.272050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.273323 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.273326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.276172 (+ 2846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.276174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.276274 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.276278 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:47:12.276335 (+ 57us) client_negotiation.cc:240] Negotiation successful
0807 22:47:12.276380 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260807 22:47:12.277242 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.270402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:54053 (local address 127.16.98.131:41469)
0807 22:47:12.270446 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0807 22:47:12.270448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:47:12.270457 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:47:12.270488 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:47:12.270489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:47:12.270498 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:47:12.270554 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:47:12.271210 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.271914 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.275108 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.275701 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.275854 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.275937 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 22:47:12.277132 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0807 22:47:12.277181 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260807 22:47:12.277562 28757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ca6375b0b7483c9ec4f735a03c9ae0" candidate_uuid: "dfd5114296a5416b8af8ca70b1e8f985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9058e666bc1475dbecf38e418326a78" is_pre_election: true
I20260807 22:47:12.277727 28757 raft_consensus.cc:2454] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfd5114296a5416b8af8ca70b1e8f985 in term 0.
I20260807 22:47:12.277987 28555 leader_election.cc:302] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9058e666bc1475dbecf38e418326a78, dfd5114296a5416b8af8ca70b1e8f985; no voters:
I20260807 22:47:12.278117 28829 raft_consensus.cc:2781] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:47:12.278155 28829 raft_consensus.cc:482] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:47:12.278174 28829 raft_consensus.cc:3037] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:47:12.279155 28829 raft_consensus.cc:504] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff12597aad496694007454d68e99a6" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 35831 } } peers { permanent_uuid: "dfd5114296a5416b8af8ca70b1e8f985" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38071 } } peers { permanent_uuid: "a9058e666bc1475dbecf38e418326a78" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41469 } }
I20260807 22:47:12.279294 28829 leader_election.cc:288] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [CANDIDATE]: Term 1 election: Requested vote from peers 33ff12597aad496694007454d68e99a6 (127.16.98.129:35831), a9058e666bc1475dbecf38e418326a78 (127.16.98.131:41469)
I20260807 22:47:12.279529 28757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ca6375b0b7483c9ec4f735a03c9ae0" candidate_uuid: "dfd5114296a5416b8af8ca70b1e8f985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9058e666bc1475dbecf38e418326a78"
I20260807 22:47:12.279596 28757 raft_consensus.cc:3037] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:47:12.280463 28757 raft_consensus.cc:2454] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfd5114296a5416b8af8ca70b1e8f985 in term 1.
I20260807 22:47:12.280661 28555 leader_election.cc:302] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9058e666bc1475dbecf38e418326a78, dfd5114296a5416b8af8ca70b1e8f985; no voters:
I20260807 22:47:12.280745 28829 raft_consensus.cc:2781] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:47:12.280885 28829 raft_consensus.cc:686] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [term 1 LEADER]: Becoming Leader. State: Replica: dfd5114296a5416b8af8ca70b1e8f985, State: Running, Role: LEADER
I20260807 22:47:12.280984 28829 consensus_queue.cc:237] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff12597aad496694007454d68e99a6" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 35831 } } peers { permanent_uuid: "dfd5114296a5416b8af8ca70b1e8f985" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38071 } } peers { permanent_uuid: "a9058e666bc1475dbecf38e418326a78" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41469 } }
I20260807 22:47:12.281806 28314 catalog_manager.cc:5495] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfd5114296a5416b8af8ca70b1e8f985 (127.16.98.130). New cstate: current_term: 1 leader_uuid: "dfd5114296a5416b8af8ca70b1e8f985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff12597aad496694007454d68e99a6" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 35831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfd5114296a5416b8af8ca70b1e8f985" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9058e666bc1475dbecf38e418326a78" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41469 } health_report { overall_health: UNKNOWN } } }
I20260807 22:47:12.283023 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.270967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:53333 (local address 127.16.98.129:35831)
0807 22:47:12.271009 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0807 22:47:12.271012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:47:12.276659 (+ 5647us) server_negotiation.cc:407] Connection header received
0807 22:47:12.276685 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:47:12.276686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:47:12.276698 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:47:12.276741 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:47:12.277732 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.278383 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.279837 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.280455 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.280528 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.280537 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 22:47:12.282893 (+ 2356us) server_negotiation.cc:299] Negotiation successful
0807 22:47:12.282960 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260807 22:47:12.281386 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.270695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:35831 (local address 127.16.98.130:53333)
0807 22:47:12.276529 (+ 5834us) negotiation.cc:107] Waiting for socket to connect
0807 22:47:12.276541 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:47:12.276603 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:47:12.277605 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:47:12.277607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:47:12.277609 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:47:12.277645 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.277646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.278629 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.278630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.279658 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.279661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.281131 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.281132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.281194 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.281198 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:47:12.281253 (+ 55us) client_negotiation.cc:240] Negotiation successful
0807 22:47:12.281290 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5823,"thread_start_us":128,"threads_started":1}
I20260807 22:47:12.283674 28481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ca6375b0b7483c9ec4f735a03c9ae0" candidate_uuid: "dfd5114296a5416b8af8ca70b1e8f985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33ff12597aad496694007454d68e99a6" is_pre_election: true
I20260807 22:47:12.283776 28481 raft_consensus.cc:2454] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfd5114296a5416b8af8ca70b1e8f985 in term 0.
I20260807 22:47:12.283854 28480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ca6375b0b7483c9ec4f735a03c9ae0" candidate_uuid: "dfd5114296a5416b8af8ca70b1e8f985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33ff12597aad496694007454d68e99a6"
I20260807 22:47:12.283895 28480 raft_consensus.cc:3037] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:47:12.284782 28480 raft_consensus.cc:2454] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfd5114296a5416b8af8ca70b1e8f985 in term 1.
I20260807 22:47:12.291188 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.274568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:49125 (local address 127.16.98.131:41469)
0807 22:47:12.277314 (+ 2746us) server_negotiation.cc:206] Beginning negotiation
0807 22:47:12.277316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:47:12.285320 (+ 8004us) server_negotiation.cc:407] Connection header received
0807 22:47:12.285423 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:47:12.285424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:47:12.285435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:47:12.285464 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:47:12.285648 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.286345 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.290216 (+ 3871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.290769 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.290819 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.290827 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0807 22:47:12.291068 (+ 241us) server_negotiation.cc:299] Negotiation successful
0807 22:47:12.291111 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2732,"thread_start_us":78,"threads_started":1}
I20260807 22:47:12.291671 28836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.274319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:41469 (local address 127.16.98.129:49125)
0807 22:47:12.285245 (+ 10926us) negotiation.cc:107] Waiting for socket to connect
0807 22:47:12.285259 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 22:47:12.285351 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:47:12.285522 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:47:12.285522 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:47:12.285524 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:47:12.285583 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.285584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.286454 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.286454 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.287598 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.287600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.290901 (+ 3301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.290902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.290988 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.290993 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:47:12.291539 (+ 546us) client_negotiation.cc:240] Negotiation successful
0807 22:47:12.291587 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10905,"thread_start_us":107,"threads_started":1}
I20260807 22:47:12.292958 28757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ca6375b0b7483c9ec4f735a03c9ae0" candidate_uuid: "33ff12597aad496694007454d68e99a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9058e666bc1475dbecf38e418326a78" is_pre_election: true
I20260807 22:47:12.293081 28757 raft_consensus.cc:2379] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 33ff12597aad496694007454d68e99a6 in current term 1: Already voted for candidate dfd5114296a5416b8af8ca70b1e8f985 in this term.
I20260807 22:47:12.293602 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.274952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:36807 (local address 127.16.98.130:38071)
0807 22:47:12.275009 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0807 22:47:12.275011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:47:12.287777 (+ 12766us) server_negotiation.cc:407] Connection header received
0807 22:47:12.287919 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:47:12.287920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:47:12.287939 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:47:12.287967 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:47:12.288181 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.288873 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.291900 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.292509 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.292579 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.292587 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0807 22:47:12.293489 (+ 902us) server_negotiation.cc:299] Negotiation successful
0807 22:47:12.293535 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 22:47:12.294001 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.274774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:38071 (local address 127.16.98.129:36807)
0807 22:47:12.287719 (+ 12945us) negotiation.cc:107] Waiting for socket to connect
0807 22:47:12.287729 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:47:12.287845 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:47:12.288046 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:47:12.288046 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:47:12.288048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:47:12.288114 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.288115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.288982 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.288983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.290101 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:47:12.290104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.292647 (+ 2543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.292648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:47:12.292716 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.292719 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:47:12.293845 (+ 1126us) client_negotiation.cc:240] Negotiation successful
0807 22:47:12.293891 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12927,"thread_start_us":64,"threads_started":1}
I20260807 22:47:12.294438 28621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ca6375b0b7483c9ec4f735a03c9ae0" candidate_uuid: "33ff12597aad496694007454d68e99a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfd5114296a5416b8af8ca70b1e8f985" is_pre_election: true
I20260807 22:47:12.294803 28416 leader_election.cc:302] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33ff12597aad496694007454d68e99a6; no voters: a9058e666bc1475dbecf38e418326a78, dfd5114296a5416b8af8ca70b1e8f985
I20260807 22:47:12.295006 28828 raft_consensus.cc:2726] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 22:47:12.305286 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.301952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34914 (local address 127.16.98.130:38071)
0807 22:47:12.301996 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0807 22:47:12.301998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:47:12.302008 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:47:12.302104 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:47:12.302105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:47:12.302128 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:47:12.302197 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:47:12.302397 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.303102 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.304188 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:47:12.304447 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:47:12.304505 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:47:12.304738 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:47:12.304871 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:47:12.304953 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 22:47:12.305167 (+ 214us) server_negotiation.cc:299] Negotiation successful
0807 22:47:12.305188 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 22:47:12.307868 28757 raft_consensus.cc:1261] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78 [term 1 FOLLOWER]: Refusing update from remote peer dfd5114296a5416b8af8ca70b1e8f985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:47:12.308248 28837 consensus_queue.cc:1035] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9058e666bc1475dbecf38e418326a78" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:47:12.310899 28480 raft_consensus.cc:1261] T b9ca6375b0b7483c9ec4f735a03c9ae0 P 33ff12597aad496694007454d68e99a6 [term 1 FOLLOWER]: Refusing update from remote peer dfd5114296a5416b8af8ca70b1e8f985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:47:12.311295 28837 consensus_queue.cc:1035] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ff12597aad496694007454d68e99a6" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:47:12.315076 28840 mvcc.cc:205] Tried to move back new op lower bound from 7316041041130094592 to 7316041041023488000. Current Snapshot: MvccSnapshot[applied={T|T < 7316041041130094592}]
I20260807 22:47:12.316041 28843 mvcc.cc:205] Tried to move back new op lower bound from 7316041041130094592 to 7316041041023488000. Current Snapshot: MvccSnapshot[applied={T|T < 7316041041130094592}]
I20260807 22:47:12.319108 28841 mvcc.cc:205] Tried to move back new op lower bound from 7316041041130094592 to 7316041041023488000. Current Snapshot: MvccSnapshot[applied={T|T < 7316041041130094592}]
I20260807 22:47:12.323410 28314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:47:12.323539 28314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:47:12.324468 28314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e371c3dc1e2464fb3a43be666094d08: Sending DeleteTablet for 3 replicas of tablet b9ca6375b0b7483c9ec4f735a03c9ae0
I20260807 22:47:12.324990 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ca6375b0b7483c9ec4f735a03c9ae0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:47:12 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:42652
I20260807 22:47:12.325002 28736 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ca6375b0b7483c9ec4f735a03c9ae0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:47:12 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:42464
I20260807 22:47:12.325490 28601 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ca6375b0b7483c9ec4f735a03c9ae0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:47:12 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:34912
I20260807 22:47:12.326120 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 28396
I20260807 22:47:12.333171 28851 tablet_replica.cc:330] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985: stopping tablet replica
I20260807 22:47:12.333402 28851 raft_consensus.cc:2229] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:47:12.333463 28850 tablet_replica.cc:330] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78: stopping tablet replica
I20260807 22:47:12.333578 28850 raft_consensus.cc:2229] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:47:12.333710 28850 raft_consensus.cc:2258] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:47:12.333786 28851 raft_consensus.cc:2258] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:47:12.334394 28851 ts_tablet_manager.cc:1905] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:47:12.336375 28850 ts_tablet_manager.cc:1905] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:47:12.337988 28851 ts_tablet_manager.cc:1918] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:47:12.338719 28851 log.cc:1196] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/b9ca6375b0b7483c9ec4f735a03c9ae0
I20260807 22:47:12.339123 28851 ts_tablet_manager.cc:1939] T b9ca6375b0b7483c9ec4f735a03c9ae0 P dfd5114296a5416b8af8ca70b1e8f985: Deleting consensus metadata
I20260807 22:47:12.341684 28301 catalog_manager.cc:4841] TS dfd5114296a5416b8af8ca70b1e8f985 (127.16.98.130:38071): tablet b9ca6375b0b7483c9ec4f735a03c9ae0 (table test-table [id=efe6836955944a0e96b085cb0f802457]) successfully deleted
I20260807 22:47:12.344341 28850 ts_tablet_manager.cc:1918] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:47:12.344450 28850 log.cc:1196] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/b9ca6375b0b7483c9ec4f735a03c9ae0
I20260807 22:47:12.345118 28850 ts_tablet_manager.cc:1939] T b9ca6375b0b7483c9ec4f735a03c9ae0 P a9058e666bc1475dbecf38e418326a78: Deleting consensus metadata
W20260807 22:47:12.345528 28301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:35831 (error 108)
I20260807 22:47:12.345630 28300 catalog_manager.cc:4841] TS a9058e666bc1475dbecf38e418326a78 (127.16.98.131:41469): tablet b9ca6375b0b7483c9ec4f735a03c9ae0 (table test-table [id=efe6836955944a0e96b085cb0f802457]) successfully deleted
I20260807 22:47:12.352921 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 28533
I20260807 22:47:12.358034 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:47:12.357874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:35831 (local address 127.0.0.1:42664)
0807 22:47:12.357941 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0807 22:47:12.357987 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:35831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 22:47:12.358317 28301 catalog_manager.cc:4584] TS 33ff12597aad496694007454d68e99a6 (127.16.98.129:35831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9ca6375b0b7483c9ec4f735a03c9ae0: Network error: Client connection negotiation failed: client connection to 127.16.98.129:35831: connect: Connection refused (error 111)
I20260807 22:47:12.364379 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 28674
I20260807 22:47:12.375090 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 28281
I20260807 22:47:12.385879 16778 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 22:47:12.732221 16778 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 22:47:12.732345 16778 mini_ranger.cc:66] Stopping Ranger...
I20260807 22:47:13.087751 16778 mini_ranger.cc:68] Stopped Ranger
2026-08-07 22:47:13.088 UTC [27860] LOG: received smart shutdown request
2026-08-07 22:47:13.093 UTC [27860] LOG: background worker "logical replication launcher" (PID 27870) exited with exit code 1
2026-08-07 22:47:13.094 UTC [27864] LOG: shutting down
2026-08-07 22:47:13.314 UTC [27860] LOG: database system is shut down
2026-08-07T22:47:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (55135 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:47:13 dist-test-slave-rgb1 krb5kdc[28861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:47:13 dist-test-slave-rgb1 krb5kdc[28861](info): set up 2 sockets
Aug 07 22:47:13 dist-test-slave-rgb1 krb5kdc[28861](info): commencing operation
krb5kdc: starting...
W20260807 22:47:14.616187 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:47:14 dist-test-slave-rgb1 krb5kdc[28861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:47:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:47:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.98.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.98.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/rangeradmin_127.16.98.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/rangeradmin_127.16.98.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.98.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.98.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/rangerlookup_127.16.98.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/rangerlookup_127.16.98.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.98.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.98.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/HTTP_127.16.98.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/HTTP_127.16.98.148@KRBTEST.COM.keytab.
I20260807 22:47:14.820820 16778 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskG4wJ7J/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/postgres -l logfile start
2026-08-07 22:47:18.193 UTC [28891] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 22:47:18.193 UTC [28891] LOG: listening on IPv4 address "127.16.98.148", port 60443
2026-08-07 22:47:18.198 UTC [28891] LOG: listening on Unix socket "/tmp/.s.PGSQL.60443"
2026-08-07 22:47:18.237 UTC [28894] LOG: database system was shut down at 2026-08-07 22:47:15 UTC
2026-08-07 22:47:18.242 UTC [28891] LOG: database system is ready to accept connections
I20260807 22:47:19.290225 16778 mini_postgres.cc:96] Postgres bound to 60443
2026-08-07 22:47:19.298 UTC [28903] FATAL: database "slave" does not exist
127.16.98.148:60443 - accepting connections
I20260807 22:47:19.298591 16778 mini_ranger.cc:161] Starting Ranger...
I20260807 22:47:19.317541 16778 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 22:47:20.077253 16778 mini_ranger.cc:87] Created ranger Postgres database
I20260807 22:47:20.077366 16778 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-admin
2026-08-07 22:47:20,386 [I] DB FLAVOR :POSTGRES
2026-08-07 22:47:20,386 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 22:47:20,387 [I] Checking connection..
2026-08-07 22:47:20,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 22:47:20,670 [I] Checking connection passed.
2026-08-07 22:47:20,670 [I] --------- Verifying version history table ---------
2026-08-07 22:47:20,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 22:47:20,961 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 22:47:20,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 22:47:21,252 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 22:47:21,252 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 22:47:21,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 22:47:21.523 UTC [28968] WARNING: there is no transaction in progress
2026-08-07 22:47:21,531 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 22:47:21,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 22:47:21,819 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 22:47:21,819 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 22:47:21,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 22:47:22,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 22:47:22,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rgb1.c.gcp-upstream.internal','N') ;"
2026-08-07 22:47:22,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 22:47:22,907 [I] Table x_portal_user does not exist in database ranger
2026-08-07 22:47:22,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 22:47:23,183 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 22:47:23,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 22:47:23,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 22:47:23,718 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 22:47:23,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 22:47:24.877 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:24.900 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:24.925 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:24.946 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:24.970 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:24.994 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:25.074 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:25.083 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:25.101 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:25.115 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:25.131 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:25.151 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:25.162 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:25.172 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:25.184 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:25.195 UTC [29077] WARNING: there is no transaction in progress
2026-08-07 22:47:26,106 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 22:47:26,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 22:47:26,355 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 22:47:26,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 22:47:26,621 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 22:47:26,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 22:47:26,885 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 22:47:26,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 22:47:27,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 22:47:27,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rgb1.c.gcp-upstream.internal';"
2026-08-07 22:47:27,709 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 22:47:27,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 22:47:27,980 [I] DB_PATCHES have already been applied
I20260807 22:47:27.987555 16778 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-admin:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskG4wJ7J/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 22:47:27.987807 16778 mini_ranger.cc:193] Using host: 127.16.98.148
I20260807 22:47:27.993053 16778 mini_ranger.cc:227] Ranger admin URL: http://127.16.98.148:44205
Aug 07, 2026 10:47:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 10:47:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 10:47:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 10:47:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 10:47:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 10:47:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 10:47:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 22:47:29.140995 16778 mini_ranger.cc:160] Time spent starting Ranger: real 9.842s user 0.000s sys 0.014s
Aug 07, 2026 10:47:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 10:47:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 07, 2026 10:47:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 10:47:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 10:47:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 10:47:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 10:47:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 10:47:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 10:47:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 10:47:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 10:47:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 10:47:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 10:47:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 10:47:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 10:47:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 10:47:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 10:47:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 10:47:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 22:47:56.188582 16778 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.98.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.98.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/rangerkms_127.16.98.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.98.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/rangerkms_127.16.98.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.98.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.98.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.98.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/HTTP_127.16.98.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.98.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/HTTP_127.16.98.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 22:47:56 dist-test-slave-rgb1 krb5kdc[28861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142876, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 22:47:56.297194 16778 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 22:47:56.312165 16778 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 22:47:57.516045 16778 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 22:47:57.516136 16778 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-kms
I20260807 22:47:57.516151 16778 mini_ranger_kms.cc:227] Using postgres at 127.16.98.148:60443
2026-08-07 22:47:57,726 [I] DB FLAVOR :POSTGRES
2026-08-07 22:47:57,726 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 22:47:57,726 [I] Checking connection
2026-08-07 22:47:57,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 22:47:57,988 [I] connection success
2026-08-07 22:47:57,989 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 22:47:57,989 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 22:47:57,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 22:47:58,271 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 22:47:58,271 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 22:47:58,271 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 22:47:58,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.98.148:60443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 22:47:58,579 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 22:47:59.158306 16778 mini_ranger_kms.cc:310] Created kms service
I20260807 22:47:59.408996 16778 mini_ranger_kms.cc:326] Created kudu user
I20260807 22:47:59.464839 16778 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 22:47:59 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142876, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:47:59.923980 16778 mini_ranger_kms.cc:384] Added ranger policy
I20260807 22:47:59.924247 16778 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-kms:/tmp/dist-test-taskG4wJ7J/build/release/bin/postgresql.jar:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskG4wJ7J/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 22:47:59.924274 16778 mini_ranger_kms.cc:240] Using host: 127.16.98.148
I20260807 22:47:59.929514 16778 mini_ranger_kms.cc:276] Ranger KMS PID: 29262
I20260807 22:47:59.929617 16778 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.98.148:40929
Aug 07, 2026 10:48:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 10:48:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 10:48:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 10:48:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 10:48:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 10:48:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260807 22:48:02.521188 16778 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.224s user 0.000s sys 0.193s
I20260807 22:48:02.521301 16778 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 22:48:02.521347 16778 mini_ranger_kms.cc:401] 127.16.98.148:40929/kms/v1/keys
Aug 07 22:48:02 dist-test-slave-rgb1 krb5kdc[28861](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786142882, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.98.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 07 22:48:03 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142882, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.98.148@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
Aug 07 22:48:03 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142882, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.98.148@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
Aug 07 22:48:06 dist-test-slave-rgb1 krb5kdc[28861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:48:06.288640 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:34675
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:34675
--ranger_config_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.98.148:40929/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:06.381949 29312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:06.382175 29312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:06.382200 29312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:06.383438 29312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:48:06.383492 29312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:06.383507 29312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:06.383520 29312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:48:06.383533 29312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:48:06.385399 29312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.98.148:40929/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:34675
--ranger_config_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:34675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:06.385707 29312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:06.385967 29312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:06.391387 29318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:06.391389 29320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:06.391458 29312 server_base.cc:1034] running on GCE node
W20260807 22:48:06.391587 29317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:06.392066 29312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:06.392411 29312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:06.393615 29312 hybrid_clock.cc:648] HybridClock initialized: now 1786142886393585 us; error 46 us; skew 500 ppm
Aug 07 22:48:06 dist-test-slave-rgb1 krb5kdc[28861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142886, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:06.396797 29312 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:48:06.397403 29312 webserver.cc:460] Webserver started at http://127.16.98.190:41261/ using document root <none> and password file <none>
I20260807 22:48:06.397680 29312 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:06.397743 29312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:06.397904 29312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:48:06 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142886, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:48:06.449206 29312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "537c05c1b5d1475dbbad7470f20e4a18"
format_stamp: "Formatted at 2026-08-07 22:48:06 on dist-test-slave-rgb1"
server_key: "3e7b3d8411a210169eede96a5e2de49a"
server_key_iv: "03da41194025583ac29e78acac76716e"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:06.449632 29312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "537c05c1b5d1475dbbad7470f20e4a18"
format_stamp: "Formatted at 2026-08-07 22:48:06 on dist-test-slave-rgb1"
server_key: "3e7b3d8411a210169eede96a5e2de49a"
server_key_iv: "03da41194025583ac29e78acac76716e"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:06.451161 29312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:48:06.464529 29327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:06.464805 29312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:06.464933 29312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "537c05c1b5d1475dbbad7470f20e4a18"
format_stamp: "Formatted at 2026-08-07 22:48:06 on dist-test-slave-rgb1"
server_key: "3e7b3d8411a210169eede96a5e2de49a"
server_key_iv: "03da41194025583ac29e78acac76716e"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:06.465030 29312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:06.482093 29312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:06.484439 29312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:06.484656 29312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:06.489351 29312 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:34675
I20260807 22:48:06.489392 29381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:34675 every 8 connection(s)
I20260807 22:48:06.489766 29312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:48:06.490538 29382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:06.493045 29382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18: Bootstrap starting.
I20260807 22:48:06.493693 29382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:06.494020 29382 log.cc:824] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:06.494925 29382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18: No bootstrap required, opened a new log
I20260807 22:48:06.496678 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 29312
I20260807 22:48:06.496821 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:48:06.497056 16778 external_mini_cluster.cc:1428] Setting key 145117ae3b883a3cb4c7c3407407ceb0
I20260807 22:48:06.497514 29382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537c05c1b5d1475dbbad7470f20e4a18" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34675 } }
I20260807 22:48:06.497782 29382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:06.497871 29382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 537c05c1b5d1475dbbad7470f20e4a18, State: Initialized, Role: FOLLOWER
I20260807 22:48:06.498023 29382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537c05c1b5d1475dbbad7470f20e4a18" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34675 } }
I20260807 22:48:06.498111 29382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:48:06.498142 29382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:48:06.498191 29382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:06.498840 29382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537c05c1b5d1475dbbad7470f20e4a18" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34675 } }
I20260807 22:48:06.498982 29382 leader_election.cc:302] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 537c05c1b5d1475dbbad7470f20e4a18; no voters:
I20260807 22:48:06.499181 29382 leader_election.cc:288] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:48:06.499267 29388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:48:06.499454 29382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:48:06.499490 29388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 1 LEADER]: Becoming Leader. State: Replica: 537c05c1b5d1475dbbad7470f20e4a18, State: Running, Role: LEADER
I20260807 22:48:06.500694 29388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537c05c1b5d1475dbbad7470f20e4a18" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34675 } }
I20260807 22:48:06.501659 29389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "537c05c1b5d1475dbbad7470f20e4a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537c05c1b5d1475dbbad7470f20e4a18" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34675 } } }
I20260807 22:48:06.501713 29382 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 22:48:06.501691 29390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 537c05c1b5d1475dbbad7470f20e4a18. Latest consensus state: current_term: 1 leader_uuid: "537c05c1b5d1475dbbad7470f20e4a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537c05c1b5d1475dbbad7470f20e4a18" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34675 } } }
I20260807 22:48:06.501760 29389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:06.501791 29390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:06.501964 29402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:48:06.505813 29402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 07 22:48:06 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:06.510776 29402 catalog_manager.cc:1347] Generated new cluster ID: 330e434427f1492398d24b4a560c92f8
I20260807 22:48:06.510862 29402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:48:06.513757 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:06.497659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.16.98.190:34675)
0807 22:48:06.497872 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:06.497883 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:06.497919 (+ 36us) server_negotiation.cc:407] Connection header received
0807 22:48:06.498042 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:06.498050 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:06.498110 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:06.498277 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:06.498942 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:06.500803 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:06.502261 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:06.505791 (+ 3530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:06.506064 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:06.508983 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:06.508988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:06.508989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:06.509003 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:06.511363 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:06.511848 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:06.511851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:06.511851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:06.511964 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:06.512197 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:06.512198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:06.512199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:06.512518 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:06.513189 (+ 671us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:06.513272 (+ 83us) server_negotiation.cc:299] Negotiation successful
0807 22:48:06.513453 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":104,"threads_started":1}
I20260807 22:48:06.537570 29402 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:48:06.538728 29402 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:48:06.550840 29402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18: Generated new TSK 0
I20260807 22:48:06.551087 29402 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:48:07.289719 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.98.148:40929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34675
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:07.385722 29429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:07.385977 29429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:07.386018 29429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:07.388057 29429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:07.388180 29429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:07.388230 29429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:48:07.390354 29429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.98.148:40929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:07.390679 29429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:07.390964 29429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:07.396445 29434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:07.396616 29437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:07.396668 29435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:07.397832 29429 server_base.cc:1034] running on GCE node
I20260807 22:48:07.398116 29429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:07.398460 29429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:07.399646 29429 hybrid_clock.cc:648] HybridClock initialized: now 1786142887399625 us; error 36 us; skew 500 ppm
Aug 07 22:48:07 dist-test-slave-rgb1 krb5kdc[28861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:07.403290 29429 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:48:07.403930 29429 webserver.cc:460] Webserver started at http://127.16.98.129:35109/ using document root <none> and password file <none>
I20260807 22:48:07.404286 29429 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:07.404413 29429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:07.404625 29429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:48:07 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:48:07.429934 29429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "6b6ff915e24e4cf09e5413130b6e86b3"
format_stamp: "Formatted at 2026-08-07 22:48:07 on dist-test-slave-rgb1"
server_key: "21d721094d3ed327c25a1f5d44bc778e"
server_key_iv: "f19a8b51caec4f88d45a0277e517cd31"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:07.430368 29429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "6b6ff915e24e4cf09e5413130b6e86b3"
format_stamp: "Formatted at 2026-08-07 22:48:07 on dist-test-slave-rgb1"
server_key: "21d721094d3ed327c25a1f5d44bc778e"
server_key_iv: "f19a8b51caec4f88d45a0277e517cd31"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:07.432071 29429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:48:07.446530 29445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:07.446781 29429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:07.446898 29429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "6b6ff915e24e4cf09e5413130b6e86b3"
format_stamp: "Formatted at 2026-08-07 22:48:07 on dist-test-slave-rgb1"
server_key: "21d721094d3ed327c25a1f5d44bc778e"
server_key_iv: "f19a8b51caec4f88d45a0277e517cd31"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:07.447098 29429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:07.477756 29429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:07.481482 29429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:07.481673 29429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:07.481954 29429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:07.482365 29429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:07.482417 29429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:07.482452 29429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:07.482481 29429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:07.490762 29429 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:39989
I20260807 22:48:07.491215 29429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:48:07.493872 29563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:39989 every 8 connection(s)
I20260807 22:48:07.497983 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 29429
I20260807 22:48:07.498116 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:48:07.498318 16778 external_mini_cluster.cc:1428] Setting key 0bfd0b236714f90de87035776e965da4
Aug 07 22:48:07 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:07.509665 29567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:07.494728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:34675 (local address 127.16.98.129:59287)
0807 22:48:07.496967 (+ 2239us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:07.497026 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:07.498622 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:07.499090 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:07.499094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:07.499389 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:07.499752 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:07.499760 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:07.500864 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:07.500865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:07.502188 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:07.502191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:07.502798 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:07.502799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:07.502956 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:07.503659 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:07.503672 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:07.505549 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:07.507801 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:07.507810 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:07.507813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:07.508151 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:07.508407 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:07.508408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:07.508408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:07.508595 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:07.508925 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:07.508932 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:07.509115 (+ 183us) client_negotiation.cc:769] Sending connection context
0807 22:48:07.509219 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 22:48:07.509403 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2181,"thread_start_us":108,"threads_started":1}
I20260807 22:48:07.510418 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:07.494719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:59287 (local address 127.16.98.190:34675)
0807 22:48:07.495020 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:07.495023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:07.497161 (+ 2138us) server_negotiation.cc:407] Connection header received
0807 22:48:07.498817 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:07.498819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:07.498865 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:07.498938 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:07.499879 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:07.500746 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:07.502290 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:07.502639 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:07.502705 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:07.505678 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:07.505680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:07.505681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:07.505690 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:07.507689 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:07.508246 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:07.508247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:07.508248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:07.508319 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:07.508679 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:07.508680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:07.508680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:07.508836 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:07.508897 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:07.510189 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0807 22:48:07.510292 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":110,"threads_started":1}
I20260807 22:48:07.511333 29564 heartbeater.cc:347] Connected to a master server at 127.16.98.190:34675
I20260807 22:48:07.511498 29564 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:07.511786 29564 heartbeater.cc:511] Master 127.16.98.190:34675 requested a full tablet report, sending...
I20260807 22:48:07.512440 29346 ts_manager.cc:194] Registered new tserver with Master: 6b6ff915e24e4cf09e5413130b6e86b3 (127.16.98.129:39989)
I20260807 22:48:07.513375 29346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:59287
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:48:07.545913 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.98.148:40929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34675
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:07.687197 29570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:07.687577 29570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:07.687668 29570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:07.689739 29570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:07.689882 29570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:07.689971 29570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:48:07.692880 29570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.98.148:40929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:07.693429 29570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:07.693831 29570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 07 22:48:07 dist-test-slave-rgb1 krb5kdc[28861](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786142887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 22:48:07.703249 29579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:07.703500 29578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:07.703581 29582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:07.704270 29570 server_base.cc:1034] running on GCE node
I20260807 22:48:07.704838 29570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:07.705142 29570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:07.706338 29570 hybrid_clock.cc:648] HybridClock initialized: now 1786142887706314 us; error 38 us; skew 500 ppm
Aug 07 22:48:07 dist-test-slave-rgb1 krb5kdc[28861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:07.709664 29570 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:48:07.710206 29570 webserver.cc:460] Webserver started at http://127.16.98.130:42167/ using document root <none> and password file <none>
I20260807 22:48:07.710472 29570 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:07.710525 29570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:07.710738 29570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:48:07 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:48:07.729631 29570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "6bcbc13c64c9462bb899341481f34769"
format_stamp: "Formatted at 2026-08-07 22:48:07 on dist-test-slave-rgb1"
server_key: "79dc9f07e6b90fd2f14d81aa93e3bea9"
server_key_iv: "3b0208554f1bd2919cd79a06185fac64"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:07.730147 29570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "6bcbc13c64c9462bb899341481f34769"
format_stamp: "Formatted at 2026-08-07 22:48:07 on dist-test-slave-rgb1"
server_key: "79dc9f07e6b90fd2f14d81aa93e3bea9"
server_key_iv: "3b0208554f1bd2919cd79a06185fac64"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:07.747213 29570 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.002s sys 0.000s
I20260807 22:48:07.769470 29589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:07.769799 29570 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260807 22:48:07.769908 29570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "6bcbc13c64c9462bb899341481f34769"
format_stamp: "Formatted at 2026-08-07 22:48:07 on dist-test-slave-rgb1"
server_key: "79dc9f07e6b90fd2f14d81aa93e3bea9"
server_key_iv: "3b0208554f1bd2919cd79a06185fac64"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:07.770100 29570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:07.794096 29570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:07.796759 29570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:07.796995 29570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:07.797258 29570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:07.797693 29570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:07.797798 29570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:07.797868 29570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:07.797923 29570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:07.804569 29570 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:44067
I20260807 22:48:07.805068 29570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:48:07.810019 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 29570
I20260807 22:48:07.810129 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:48:07.810328 16778 external_mini_cluster.cc:1428] Setting key 53f6b52dcc9325f8db67ab80b9c99483
I20260807 22:48:07.807862 29704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:44067 every 8 connection(s)
Aug 07 22:48:07 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:07.824617 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:07.808464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:34675 (local address 127.16.98.130:42641)
0807 22:48:07.809018 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:07.809063 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:07.809727 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:07.813749 (+ 4022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:07.813754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:07.814026 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:07.814326 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:07.814332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:07.815367 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:07.815368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:07.816634 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:07.816638 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:07.817276 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:07.817277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:07.817461 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:07.818267 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:07.818278 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:07.819954 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:07.822407 (+ 2453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:07.822418 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:07.822436 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:07.822824 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:07.823087 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:07.823088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:07.823089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:07.823260 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:07.823666 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:07.823674 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:07.823888 (+ 214us) client_negotiation.cc:769] Sending connection context
0807 22:48:07.824019 (+ 131us) client_negotiation.cc:240] Negotiation successful
0807 22:48:07.824223 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":136,"threads_started":1}
I20260807 22:48:07.825192 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:07.808835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:42641 (local address 127.16.98.190:34675)
0807 22:48:07.808905 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:07.808907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:07.809148 (+ 241us) server_negotiation.cc:407] Connection header received
0807 22:48:07.813468 (+ 4320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:07.813470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:07.813490 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:07.813524 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:07.814444 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:07.815247 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:07.816740 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:07.817119 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:07.817164 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:07.820082 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:07.820084 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:07.820084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:07.820093 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:07.822284 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:07.822927 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:07.822929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:07.822929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:07.823000 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:07.823348 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:07.823349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:07.823349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:07.823552 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:07.823630 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:07.824935 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0807 22:48:07.825055 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260807 22:48:07.825625 29705 heartbeater.cc:347] Connected to a master server at 127.16.98.190:34675
I20260807 22:48:07.825745 29705 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:07.825987 29705 heartbeater.cc:511] Master 127.16.98.190:34675 requested a full tablet report, sending...
I20260807 22:48:07.826395 29346 ts_manager.cc:194] Registered new tserver with Master: 6bcbc13c64c9462bb899341481f34769 (127.16.98.130:44067)
I20260807 22:48:07.826882 29346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:42641
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:48:07.868407 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.98.148:40929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34675
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:08.022112 29711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:08.022549 29711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:08.022651 29711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:08.025241 29711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:08.025398 29711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:08.025525 29711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:48:08.028441 29711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.98.148:40929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:34675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:08.029243 29711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:08.029773 29711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:08.037577 29717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:08.037565 29719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:08.037570 29716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:08.038086 29711 server_base.cc:1034] running on GCE node
I20260807 22:48:08.038324 29711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:08.038621 29711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:08.039810 29711 hybrid_clock.cc:648] HybridClock initialized: now 1786142888039778 us; error 47 us; skew 500 ppm
Aug 07 22:48:08 dist-test-slave-rgb1 krb5kdc[28861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:08.043840 29711 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:48:08.044488 29711 webserver.cc:460] Webserver started at http://127.16.98.131:45821/ using document root <none> and password file <none>
I20260807 22:48:08.044798 29711 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:08.044870 29711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:08.045025 29711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 22:48:08 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:48:08.060662 29711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "e560779109c34f28923d6758135495ac"
format_stamp: "Formatted at 2026-08-07 22:48:08 on dist-test-slave-rgb1"
server_key: "171bccd016ceb2175659dc97aba944e2"
server_key_iv: "f977699f8f046b27c55bb0bbfecce8cb"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:08.061532 29711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "e560779109c34f28923d6758135495ac"
format_stamp: "Formatted at 2026-08-07 22:48:08 on dist-test-slave-rgb1"
server_key: "171bccd016ceb2175659dc97aba944e2"
server_key_iv: "f977699f8f046b27c55bb0bbfecce8cb"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:08.069198 29711 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.001s
I20260807 22:48:08.095935 29726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:08.096180 29711 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260807 22:48:08.096290 29711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "e560779109c34f28923d6758135495ac"
format_stamp: "Formatted at 2026-08-07 22:48:08 on dist-test-slave-rgb1"
server_key: "171bccd016ceb2175659dc97aba944e2"
server_key_iv: "f977699f8f046b27c55bb0bbfecce8cb"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:08.096416 29711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:08.110805 29711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:08.113859 29711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:08.114048 29711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:08.114372 29711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:08.114809 29711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:08.114868 29711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:08.114904 29711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:08.114931 29711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:08.124200 29711 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:42625
I20260807 22:48:08.124291 29841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:42625 every 8 connection(s)
I20260807 22:48:08.124917 29711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:48:08.128875 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 29711
I20260807 22:48:08.129022 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:48:08.129249 16778 external_mini_cluster.cc:1428] Setting key 3d31e6fa3ce4983d7c73f6bd81836ec8
Aug 07 22:48:08 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:08.142607 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:08.130410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:38481 (local address 127.16.98.190:34675)
0807 22:48:08.130463 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:08.130466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:08.130885 (+ 419us) server_negotiation.cc:407] Connection header received
0807 22:48:08.131828 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:08.131829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:08.131842 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:08.131878 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:08.132865 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:08.133661 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:08.135229 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:08.135541 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:08.135593 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:08.138674 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:08.138676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:08.138676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:08.138685 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:08.140643 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:08.141294 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:08.141295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:08.141296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:08.141376 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:08.141741 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:08.141743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:08.141743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:08.141892 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:08.141997 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:08.142393 (+ 396us) server_negotiation.cc:299] Negotiation successful
0807 22:48:08.142512 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 22:48:08.142839 29844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:08.130424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:34675 (local address 127.16.98.131:38481)
0807 22:48:08.130710 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:08.130773 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:08.131683 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:08.132091 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:08.132096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:08.132396 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:08.132743 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:08.132750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:08.133796 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:08.133797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:08.135112 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:08.135115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:08.135716 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:08.135717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:08.135862 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:08.136739 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:08.136749 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:08.138540 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:08.140765 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:08.140775 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:08.140778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:08.141181 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:08.141473 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:08.141474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:08.141474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:08.141658 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:08.141990 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:08.141998 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:08.142198 (+ 200us) client_negotiation.cc:769] Sending connection context
0807 22:48:08.142345 (+ 147us) client_negotiation.cc:240] Negotiation successful
0807 22:48:08.142558 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":161,"threads_started":1}
I20260807 22:48:08.143553 29842 heartbeater.cc:347] Connected to a master server at 127.16.98.190:34675
I20260807 22:48:08.143716 29842 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:08.143973 29842 heartbeater.cc:511] Master 127.16.98.190:34675 requested a full tablet report, sending...
I20260807 22:48:08.144510 29346 ts_manager.cc:194] Registered new tserver with Master: e560779109c34f28923d6758135495ac (127.16.98.131:42625)
I20260807 22:48:08.145025 29346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:38481
I20260807 22:48:08.152536 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:48:08.154258 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 29429
I20260807 22:48:08.162041 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 29570
I20260807 22:48:08.170162 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 29711
I20260807 22:48:08.178279 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 29312
I20260807 22:48:08.187541 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:34675
--webserver_interface=127.16.98.190
--webserver_port=41261
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:34675
--ranger_config_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.98.148:40929/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.98.148:40929/kms with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:08.287536 29845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:08.287823 29845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:08.287853 29845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:08.289469 29845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:48:08.289541 29845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:08.289556 29845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:08.289570 29845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:48:08.289583 29845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:48:08.291471 29845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.98.148:40929/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:34675
--ranger_config_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:34675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=41261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:08.291764 29845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:08.292061 29845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:08.297966 29851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:08.297986 29850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:08.298002 29853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:08.298467 29845 server_base.cc:1034] running on GCE node
I20260807 22:48:08.298816 29845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:08.299178 29845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:08.300401 29845 hybrid_clock.cc:648] HybridClock initialized: now 1786142888300355 us; error 72 us; skew 500 ppm
Aug 07 22:48:08 dist-test-slave-rgb1 krb5kdc[28861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:08.303431 29845 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:48:08.303923 29845 webserver.cc:460] Webserver started at http://127.16.98.190:41261/ using document root <none> and password file <none>
I20260807 22:48:08.304211 29845 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:08.304293 29845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 22:48:08 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:48:08.397396 29845 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:48:08.398500 29861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:08.398787 29845 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:48:08.398878 29845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "537c05c1b5d1475dbbad7470f20e4a18"
format_stamp: "Formatted at 2026-08-07 22:48:06 on dist-test-slave-rgb1"
server_key: "3e7b3d8411a210169eede96a5e2de49a"
server_key_iv: "03da41194025583ac29e78acac76716e"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:08.399303 29845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:08.441277 29845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:08.444759 29845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:08.444904 29845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:08.450086 29845 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:34675
I20260807 22:48:08.450147 29915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:34675 every 8 connection(s)
I20260807 22:48:08.450558 29845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:48:08.451967 29916 sys_catalog.cc:263] Verifying existing consensus state
I20260807 22:48:08.452754 29916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18: Bootstrap starting.
I20260807 22:48:08.457196 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 29845
I20260807 22:48:08.457577 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.98.148:40929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:39989
--local_ip_for_outbound_sockets=127.16.98.129
--tserver_master_addrs=127.16.98.190:34675
--webserver_port=35109
--webserver_interface=127.16.98.129
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.98.148:40929/kms with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:48:08.461473 29916 log.cc:824] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:08.463596 29916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:48:08.463965 29916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18: Bootstrap complete.
I20260807 22:48:08.466960 29916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537c05c1b5d1475dbbad7470f20e4a18" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34675 } }
I20260807 22:48:08.467303 29916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 537c05c1b5d1475dbbad7470f20e4a18, State: Initialized, Role: FOLLOWER
I20260807 22:48:08.467552 29916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537c05c1b5d1475dbbad7470f20e4a18" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34675 } }
I20260807 22:48:08.467667 29916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:48:08.467733 29916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:48:08.467790 29916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:48:08.469129 29916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537c05c1b5d1475dbbad7470f20e4a18" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34675 } }
I20260807 22:48:08.469275 29916 leader_election.cc:302] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 537c05c1b5d1475dbbad7470f20e4a18; no voters:
I20260807 22:48:08.469640 29916 leader_election.cc:288] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 22:48:08.469762 29920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 22:48:08.470068 29920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [term 2 LEADER]: Becoming Leader. State: Replica: 537c05c1b5d1475dbbad7470f20e4a18, State: Running, Role: LEADER
I20260807 22:48:08.470090 29916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:48:08.470329 29920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537c05c1b5d1475dbbad7470f20e4a18" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34675 } }
I20260807 22:48:08.470911 29921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "537c05c1b5d1475dbbad7470f20e4a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537c05c1b5d1475dbbad7470f20e4a18" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34675 } } }
I20260807 22:48:08.471045 29921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:08.471244 29922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 537c05c1b5d1475dbbad7470f20e4a18. Latest consensus state: current_term: 2 leader_uuid: "537c05c1b5d1475dbbad7470f20e4a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537c05c1b5d1475dbbad7470f20e4a18" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 34675 } } }
I20260807 22:48:08.471314 29922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18 [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:08.472078 29929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:48:08.472688 29916 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 22:48:08.472744 29929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:48:08.472890 29929 catalog_manager.cc:1259] Loaded cluster ID: 330e434427f1492398d24b4a560c92f8
I20260807 22:48:08.472930 29929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:48:08.474407 29929 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:48:08.474638 29929 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18: Loaded TSK: 0
I20260807 22:48:08.474874 29929 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 22:48:08.601121 29918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:08.601743 29918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:08.601789 29918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:08.604115 29918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:08.604199 29918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:08.604254 29918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:48:08.607389 29918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.98.148:40929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:39989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=35109
--tserver_master_addrs=127.16.98.190:34675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:08.607772 29918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:08.608122 29918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:08.615543 29954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:08.615840 29956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:08.615981 29918 server_base.cc:1034] running on GCE node
W20260807 22:48:08.616097 29953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:08.616407 29918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:08.616791 29918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:08.617976 29918 hybrid_clock.cc:648] HybridClock initialized: now 1786142888617944 us; error 51 us; skew 500 ppm
Aug 07 22:48:08 dist-test-slave-rgb1 krb5kdc[28861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:08.624743 29918 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:48:08.625357 29918 webserver.cc:460] Webserver started at http://127.16.98.129:35109/ using document root <none> and password file <none>
I20260807 22:48:08.625624 29918 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:08.625701 29918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 22:48:08 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:48:08.644136 29918 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:08.645288 29964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:08.645622 29918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:08.645752 29918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "6b6ff915e24e4cf09e5413130b6e86b3"
format_stamp: "Formatted at 2026-08-07 22:48:07 on dist-test-slave-rgb1"
server_key: "21d721094d3ed327c25a1f5d44bc778e"
server_key_iv: "f19a8b51caec4f88d45a0277e517cd31"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:08.646310 29918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:08.690438 29918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:08.694407 29918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:08.694622 29918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:08.694943 29918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:08.695488 29918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:08.695550 29918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:08.695588 29918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:08.695608 29918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:08.704453 29918 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:39989
I20260807 22:48:08.705103 29918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:48:08.705909 30081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:39989 every 8 connection(s)
I20260807 22:48:08.709997 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 29918
I20260807 22:48:08.710364 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.98.148:40929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:44067
--local_ip_for_outbound_sockets=127.16.98.130
--tserver_master_addrs=127.16.98.190:34675
--webserver_port=42167
--webserver_interface=127.16.98.130
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.98.148:40929/kms with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:48:08 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:08.734977 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:08.706709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:34675 (local address 127.16.98.129:36559)
0807 22:48:08.707230 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:08.707283 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:08.707968 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:08.708837 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:08.708842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:08.709134 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:08.709451 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:08.709458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:08.711495 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:08.711496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:08.712362 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:08.712365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:08.713340 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:08.713342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:08.713497 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:08.714211 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:08.714233 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:08.716232 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:08.719452 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:08.719464 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:08.719467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:08.719940 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:08.720302 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:08.720303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:08.720304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:08.720459 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:08.732998 (+ 12539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:08.733015 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:08.734324 (+ 1309us) client_negotiation.cc:769] Sending connection context
0807 22:48:08.734466 (+ 142us) client_negotiation.cc:240] Negotiation successful
0807 22:48:08.734688 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":118,"threads_started":1}
I20260807 22:48:08.736272 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:08.706820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:36559 (local address 127.16.98.190:34675)
0807 22:48:08.708268 (+ 1448us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:08.708279 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:08.708321 (+ 42us) server_negotiation.cc:407] Connection header received
0807 22:48:08.708480 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:08.708486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:08.708548 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:08.709607 (+ 1059us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:08.709746 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:08.711350 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:08.712479 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:08.713195 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:08.713835 (+ 640us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:08.716401 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:08.716407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:08.716407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:08.716423 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:08.719313 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:08.720067 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:08.720071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:08.720072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:08.720207 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:08.720596 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:08.720597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:08.720597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:08.720982 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:08.735739 (+ 14757us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:08.735864 (+ 125us) server_negotiation.cc:299] Negotiation successful
0807 22:48:08.736049 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1379,"thread_start_us":133,"threads_started":1}
I20260807 22:48:08.737311 30082 heartbeater.cc:347] Connected to a master server at 127.16.98.190:34675
I20260807 22:48:08.737416 30082 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:08.737623 30082 heartbeater.cc:511] Master 127.16.98.190:34675 requested a full tablet report, sending...
W20260807 22:48:08.738574 30082 heartbeater.cc:644] Failed to heartbeat to 127.16.98.190:34675 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 22:48:08.738646 30082 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:08.738726 30082 heartbeater.cc:511] Master 127.16.98.190:34675 requested a full tablet report, sending...
I20260807 22:48:08.739151 30082 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:08.739233 30082 heartbeater.cc:511] Master 127.16.98.190:34675 requested a full tablet report, sending...
W20260807 22:48:08.739575 30082 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 22:48:08.832366 30087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:08.832697 30087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:08.832753 30087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:08.834697 30087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:08.834769 30087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:08.834813 30087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:48:08.837116 30087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.98.148:40929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:44067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=42167
--tserver_master_addrs=127.16.98.190:34675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:08.837378 30087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:08.837658 30087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:08.844638 30092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:08.844781 30093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:08.845059 30087 server_base.cc:1034] running on GCE node
W20260807 22:48:08.844705 30095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:08.845357 30087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:08.845700 30087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:08.846909 30087 hybrid_clock.cc:648] HybridClock initialized: now 1786142888846880 us; error 46 us; skew 500 ppm
Aug 07 22:48:08 dist-test-slave-rgb1 krb5kdc[28861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:08.850078 30087 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:48:08.850641 30087 webserver.cc:460] Webserver started at http://127.16.98.130:42167/ using document root <none> and password file <none>
I20260807 22:48:08.850924 30087 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:08.851012 30087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 22:48:08 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:48:08.870214 30087 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:08.871341 30104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:08.871610 30087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:08.871726 30087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "6bcbc13c64c9462bb899341481f34769"
format_stamp: "Formatted at 2026-08-07 22:48:07 on dist-test-slave-rgb1"
server_key: "79dc9f07e6b90fd2f14d81aa93e3bea9"
server_key_iv: "3b0208554f1bd2919cd79a06185fac64"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:08.872269 30087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:08.892552 30087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:08.895313 30087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:08.895514 30087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:08.895828 30087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:08.896243 30087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:08.896304 30087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:08.896345 30087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:08.896364 30087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:08.904223 30087 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:44067
I20260807 22:48:08.904299 30218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:44067 every 8 connection(s)
I20260807 22:48:08.904982 30087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:48:08.910982 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 30087
I20260807 22:48:08.911403 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.98.148:40929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:42625
--local_ip_for_outbound_sockets=127.16.98.131
--tserver_master_addrs=127.16.98.190:34675
--webserver_port=45821
--webserver_interface=127.16.98.131
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.98.148:40929/kms with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:48:08 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:08.917693 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:08.906289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:40593 (local address 127.16.98.190:34675)
0807 22:48:08.906366 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:08.906369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:08.906601 (+ 232us) server_negotiation.cc:407] Connection header received
0807 22:48:08.907154 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:08.907156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:08.907171 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:08.907263 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:08.907868 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:08.908729 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:08.910089 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:08.910400 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:08.910499 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:08.913648 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:08.913650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:08.913651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:08.913661 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:08.915584 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:08.916213 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:08.916215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:08.916216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:08.916294 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:08.916655 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:08.916657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:08.916657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:08.916890 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:08.916973 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:08.917395 (+ 422us) server_negotiation.cc:299] Negotiation successful
0807 22:48:08.917524 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 22:48:08.917871 30221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:08.906261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:34675 (local address 127.16.98.130:40593)
0807 22:48:08.906450 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:08.906498 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:08.907044 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:08.907321 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:08.907325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:08.907531 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:08.907742 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:08.907747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:08.908872 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:08.908874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:08.909981 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:08.909985 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:08.910601 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:08.910602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:08.910712 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:08.911554 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:08.911566 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:08.913406 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:08.915710 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:08.915719 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:08.915722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:08.916108 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:08.916386 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:08.916387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:08.916388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:08.916585 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:08.917006 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:08.917014 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:08.917205 (+ 191us) client_negotiation.cc:769] Sending connection context
0807 22:48:08.917340 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 22:48:08.917544 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
I20260807 22:48:08.918565 30219 heartbeater.cc:347] Connected to a master server at 127.16.98.190:34675
I20260807 22:48:08.918689 30219 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:08.918942 30219 heartbeater.cc:511] Master 127.16.98.190:34675 requested a full tablet report, sending...
W20260807 22:48:08.919701 30219 heartbeater.cc:644] Failed to heartbeat to 127.16.98.190:34675 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 22:48:08.919793 30219 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:08.919883 30219 heartbeater.cc:511] Master 127.16.98.190:34675 requested a full tablet report, sending...
I20260807 22:48:08.920261 30219 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:08.920357 30219 heartbeater.cc:511] Master 127.16.98.190:34675 requested a full tablet report, sending...
W20260807 22:48:08.920756 30219 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 22:48:09.040254 30222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:09.040503 30222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:09.040549 30222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:09.042212 30222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:09.042294 30222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:09.042348 30222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:48:09.045182 30222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.98.148:40929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:42625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=45821
--tserver_master_addrs=127.16.98.190:34675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:09.045522 30222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:09.045853 30222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:09.053941 30231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:09.054554 30222 server_base.cc:1034] running on GCE node
W20260807 22:48:09.057004 30229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:09.057292 30228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:09.057669 30222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:09.058069 30222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:09.059263 30222 hybrid_clock.cc:648] HybridClock initialized: now 1786142889059175 us; error 109 us; skew 500 ppm
Aug 07 22:48:09 dist-test-slave-rgb1 krb5kdc[28861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:09.062450 30222 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:48:09.063136 30222 webserver.cc:460] Webserver started at http://127.16.98.131:45821/ using document root <none> and password file <none>
I20260807 22:48:09.063444 30222 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:09.063547 30222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 22:48:09 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:48:09.084544 30222 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260807 22:48:09.085796 30240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:09.086036 30222 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:09.086143 30222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "e560779109c34f28923d6758135495ac"
format_stamp: "Formatted at 2026-08-07 22:48:08 on dist-test-slave-rgb1"
server_key: "171bccd016ceb2175659dc97aba944e2"
server_key_iv: "f977699f8f046b27c55bb0bbfecce8cb"
server_key_version: "kuduclusterkey@0"
I20260807 22:48:09.086580 30222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:09.109557 30222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:09.112371 30222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:09.112602 30222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:09.112908 30222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:09.113302 30222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:09.113356 30222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:09.113399 30222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:09.113430 30222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:09.122658 30222 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:42625
I20260807 22:48:09.123273 30222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:48:09.124480 30355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:42625 every 8 connection(s)
Aug 07 22:48:09 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:09.132812 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 30222
I20260807 22:48:09.137413 30358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:09.125253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:34675 (local address 127.16.98.131:41959)
0807 22:48:09.125491 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:09.125545 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:09.126207 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:09.126543 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:09.126547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:09.126793 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:09.127088 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:09.127094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:09.128117 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:09.128119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:09.129104 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:09.129107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:09.129685 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:09.129686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:09.129847 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:09.130595 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:09.130604 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:09.132422 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:09.135352 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:09.135366 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:09.135370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:09.135759 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:09.136106 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:09.136107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:09.136107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:09.136294 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:09.136619 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:09.136627 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:09.136832 (+ 205us) client_negotiation.cc:769] Sending connection context
0807 22:48:09.136961 (+ 129us) client_negotiation.cc:240] Negotiation successful
0807 22:48:09.137159 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":127,"threads_started":1}
I20260807 22:48:09.137944 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:09.124816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:41959 (local address 127.16.98.190:34675)
0807 22:48:09.124913 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:09.124915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:09.125685 (+ 770us) server_negotiation.cc:407] Connection header received
0807 22:48:09.126346 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:09.126347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:09.126363 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:09.126451 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:09.127211 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:09.128000 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:09.129206 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:09.129525 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:09.129613 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:09.132589 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:09.132592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:09.132592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:09.132603 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:09.135221 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:09.135869 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:09.135870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:09.135870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:09.135939 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:09.136373 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:09.136374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:09.136374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:09.136513 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:09.136629 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:09.137801 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0807 22:48:09.137880 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 22:48:09.138535 30356 heartbeater.cc:347] Connected to a master server at 127.16.98.190:34675
I20260807 22:48:09.138639 30356 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:09.138870 30356 heartbeater.cc:511] Master 127.16.98.190:34675 requested a full tablet report, sending...
W20260807 22:48:09.139592 30356 heartbeater.cc:644] Failed to heartbeat to 127.16.98.190:34675 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 22:48:09.139657 30356 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:09.139739 30356 heartbeater.cc:511] Master 127.16.98.190:34675 requested a full tablet report, sending...
I20260807 22:48:09.140100 30356 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:09.140182 30356 heartbeater.cc:511] Master 127.16.98.190:34675 requested a full tablet report, sending...
W20260807 22:48:09.140488 30356 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 22:48:09.146170 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:09.133433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.16.98.190:34675)
0807 22:48:09.133785 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:09.133787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:09.133797 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:48:09.133820 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:09.133821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:09.133831 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:09.133940 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:09.134699 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:09.135338 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:09.136251 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:09.136654 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:09.136742 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:09.138338 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:09.138339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:09.138340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:09.138347 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:09.140584 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:09.141031 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:09.141033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:09.141034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:09.141097 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:09.144707 (+ 3610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:09.144710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:09.144710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:09.145077 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:09.145333 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:09.145959 (+ 626us) server_negotiation.cc:299] Negotiation successful
0807 22:48:09.146078 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":92,"threads_started":1}
I20260807 22:48:09.739810 30082 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:09.740020 30082 heartbeater.cc:511] Master 127.16.98.190:34675 requested a full tablet report, sending...
I20260807 22:48:09.740513 29878 ts_manager.cc:194] Registered new tserver with Master: 6b6ff915e24e4cf09e5413130b6e86b3 (127.16.98.129:39989)
I20260807 22:48:09.741312 29878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:36559
Aug 07 22:48:09 dist-test-slave-rgb1 krb5kdc[28861](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786142889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:09.920981 30219 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:09.921186 30219 heartbeater.cc:511] Master 127.16.98.190:34675 requested a full tablet report, sending...
I20260807 22:48:09.921625 29878 ts_manager.cc:194] Registered new tserver with Master: 6bcbc13c64c9462bb899341481f34769 (127.16.98.130:44067)
I20260807 22:48:09.922302 29878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:40593
I20260807 22:48:10.140671 30356 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:10.140853 30356 heartbeater.cc:511] Master 127.16.98.190:34675 requested a full tablet report, sending...
I20260807 22:48:10.141266 29878 ts_manager.cc:194] Registered new tserver with Master: e560779109c34f28923d6758135495ac (127.16.98.131:42625)
I20260807 22:48:10.141795 29878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:41959
I20260807 22:48:10.143709 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:48:10.158316 30379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:10.145028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39652 (local address 127.16.98.190:34675)
0807 22:48:10.148733 (+ 3705us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:10.148736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:10.148931 (+ 195us) server_negotiation.cc:407] Connection header received
0807 22:48:10.149211 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:10.149212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:10.149228 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:10.149331 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:10.150500 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.151415 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.152607 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.152938 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.153033 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:10.154805 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:10.154807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:10.154808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:10.154816 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:10.156923 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:10.157373 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:10.157374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:10.157375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:10.157449 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:10.157696 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:10.157697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:10.157698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:10.157851 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:10.157964 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:10.158102 (+ 138us) server_negotiation.cc:299] Negotiation successful
0807 22:48:10.158196 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3672,"thread_start_us":76,"threads_started":1}
I20260807 22:48:10.165693 29878 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:48:10.168841 29878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:48:10.194720 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:10.185896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:42625 (local address 127.0.0.1:49474)
0807 22:48:10.187350 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:10.187397 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:10.187468 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:10.188059 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:10.188063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:10.188065 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:10.188139 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:10.188140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.190470 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.190471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:10.192321 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:10.192325 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.194312 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.194313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:10.194417 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:10.194434 (+ 17us) client_negotiation.cc:769] Sending connection context
0807 22:48:10.194547 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 22:48:10.194613 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1418,"thread_start_us":92,"threads_started":1}
Aug 07 22:48:10 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:48:10.200484 30384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:10.186145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:44067 (local address 127.0.0.1:60852)
0807 22:48:10.187582 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:10.187591 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:10.187624 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:10.190003 (+ 2379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:10.190005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:10.190007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:10.190057 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:10.190059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.197892 (+ 7833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.197893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:10.199063 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:10.199066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.200214 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.200216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:10.200285 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:10.200290 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:48:10.200339 (+ 49us) client_negotiation.cc:240] Negotiation successful
0807 22:48:10.200385 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1424,"thread_start_us":68,"threads_started":1}
I20260807 22:48:10.200794 30388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:10.186646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.16.98.131:42625)
0807 22:48:10.187819 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:10.187824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:10.187835 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:48:10.187872 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:10.187874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:10.187891 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:10.187960 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:10.188219 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.190384 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.192986 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.193806 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.193885 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:10.193952 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:10.197310 (+ 3358us) server_negotiation.cc:299] Negotiation successful
0807 22:48:10.200705 (+ 3395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141,"thread_start_us":65,"threads_started":1}
I20260807 22:48:10.201637 30290 tablet_service.cc:1467] Processing CreateTablet for tablet e2ee001bc1f24291b07428d419ca6141 (DEFAULT_TABLE table=test-table [id=9d456c56736740a38c885e6ad0d457c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:48:10.202085 30290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ee001bc1f24291b07428d419ca6141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:10.204054 30389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:10.186791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60852 (local address 127.16.98.130:44067)
0807 22:48:10.189728 (+ 2937us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:10.189733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:10.189745 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:48:10.189780 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:10.189782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:10.189829 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:10.189910 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:10.190133 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.195528 (+ 5395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.199182 (+ 3654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.200027 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.200104 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:10.200173 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:10.202936 (+ 2763us) server_negotiation.cc:299] Negotiation successful
0807 22:48:10.202995 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2896,"thread_start_us":68,"threads_started":1}
I20260807 22:48:10.204679 30385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:10.186313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:39989 (local address 127.0.0.1:47706)
0807 22:48:10.187687 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:10.187695 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:10.187725 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:10.195955 (+ 8230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:10.195956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:10.195960 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:10.196009 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:10.196011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.201139 (+ 5128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.201140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:10.202753 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:10.202756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.204403 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.204404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:10.204471 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:10.204477 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 22:48:10.204521 (+ 44us) client_negotiation.cc:240] Negotiation successful
0807 22:48:10.204604 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1363,"thread_start_us":64,"threads_started":1}
I20260807 22:48:10.204823 30152 tablet_service.cc:1467] Processing CreateTablet for tablet e2ee001bc1f24291b07428d419ca6141 (DEFAULT_TABLE table=test-table [id=9d456c56736740a38c885e6ad0d457c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:48:10.205114 30387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:10.186439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.16.98.129:39989)
0807 22:48:10.195692 (+ 9253us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:10.195698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:10.195710 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:48:10.195749 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:10.195751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:10.195771 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:10.195853 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:10.196086 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.197189 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.203124 (+ 5935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.203935 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.204865 (+ 930us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:10.204946 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:10.204975 (+ 29us) server_negotiation.cc:299] Negotiation successful
0807 22:48:10.205028 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9206,"thread_start_us":66,"threads_started":1}
I20260807 22:48:10.205236 30152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ee001bc1f24291b07428d419ca6141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:10.205793 30015 tablet_service.cc:1467] Processing CreateTablet for tablet e2ee001bc1f24291b07428d419ca6141 (DEFAULT_TABLE table=test-table [id=9d456c56736740a38c885e6ad0d457c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:48:10.206127 30015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ee001bc1f24291b07428d419ca6141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:10.210105 30390 tablet_bootstrap.cc:492] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac: Bootstrap starting.
I20260807 22:48:10.210947 30390 tablet_bootstrap.cc:654] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:10.211287 30390 log.cc:824] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:10.212776 30391 tablet_bootstrap.cc:492] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769: Bootstrap starting.
I20260807 22:48:10.213677 30391 tablet_bootstrap.cc:654] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:10.213923 30392 tablet_bootstrap.cc:492] T e2ee001bc1f24291b07428d419ca6141 P 6b6ff915e24e4cf09e5413130b6e86b3: Bootstrap starting.
I20260807 22:48:10.214622 30392 tablet_bootstrap.cc:654] T e2ee001bc1f24291b07428d419ca6141 P 6b6ff915e24e4cf09e5413130b6e86b3: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:10.215062 30391 log.cc:824] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:10.215683 30392 log.cc:824] T e2ee001bc1f24291b07428d419ca6141 P 6b6ff915e24e4cf09e5413130b6e86b3: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:10.221988 30390 tablet_bootstrap.cc:492] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac: No bootstrap required, opened a new log
I20260807 22:48:10.222095 30390 ts_tablet_manager.cc:1397] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260807 22:48:10.224413 30390 raft_consensus.cc:348] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e560779109c34f28923d6758135495ac" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42625 } } peers { permanent_uuid: "6bcbc13c64c9462bb899341481f34769" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44067 } } peers { permanent_uuid: "6b6ff915e24e4cf09e5413130b6e86b3" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39989 } }
I20260807 22:48:10.224655 30390 raft_consensus.cc:374] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:10.224699 30390 raft_consensus.cc:729] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e560779109c34f28923d6758135495ac, State: Initialized, Role: FOLLOWER
I20260807 22:48:10.224843 30390 consensus_queue.cc:260] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e560779109c34f28923d6758135495ac" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42625 } } peers { permanent_uuid: "6bcbc13c64c9462bb899341481f34769" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44067 } } peers { permanent_uuid: "6b6ff915e24e4cf09e5413130b6e86b3" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39989 } }
I20260807 22:48:10.225147 30390 ts_tablet_manager.cc:1428] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:48:10.225281 30356 heartbeater.cc:503] Master 127.16.98.190:34675 was elected leader, sending a full tablet report...
I20260807 22:48:10.227710 30391 tablet_bootstrap.cc:492] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769: No bootstrap required, opened a new log
I20260807 22:48:10.227813 30391 ts_tablet_manager.cc:1397] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260807 22:48:10.230191 30391 raft_consensus.cc:348] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e560779109c34f28923d6758135495ac" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42625 } } peers { permanent_uuid: "6bcbc13c64c9462bb899341481f34769" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44067 } } peers { permanent_uuid: "6b6ff915e24e4cf09e5413130b6e86b3" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39989 } }
I20260807 22:48:10.230389 30391 raft_consensus.cc:374] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:10.230423 30391 raft_consensus.cc:729] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bcbc13c64c9462bb899341481f34769, State: Initialized, Role: FOLLOWER
I20260807 22:48:10.230556 30391 consensus_queue.cc:260] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e560779109c34f28923d6758135495ac" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42625 } } peers { permanent_uuid: "6bcbc13c64c9462bb899341481f34769" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44067 } } peers { permanent_uuid: "6b6ff915e24e4cf09e5413130b6e86b3" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39989 } }
I20260807 22:48:10.230883 30391 ts_tablet_manager.cc:1428] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:48:10.231046 30219 heartbeater.cc:503] Master 127.16.98.190:34675 was elected leader, sending a full tablet report...
I20260807 22:48:10.232188 30392 tablet_bootstrap.cc:492] T e2ee001bc1f24291b07428d419ca6141 P 6b6ff915e24e4cf09e5413130b6e86b3: No bootstrap required, opened a new log
I20260807 22:48:10.232280 30392 ts_tablet_manager.cc:1397] T e2ee001bc1f24291b07428d419ca6141 P 6b6ff915e24e4cf09e5413130b6e86b3: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.000s
I20260807 22:48:10.235062 30392 raft_consensus.cc:348] T e2ee001bc1f24291b07428d419ca6141 P 6b6ff915e24e4cf09e5413130b6e86b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e560779109c34f28923d6758135495ac" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42625 } } peers { permanent_uuid: "6bcbc13c64c9462bb899341481f34769" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44067 } } peers { permanent_uuid: "6b6ff915e24e4cf09e5413130b6e86b3" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39989 } }
I20260807 22:48:10.235285 30392 raft_consensus.cc:374] T e2ee001bc1f24291b07428d419ca6141 P 6b6ff915e24e4cf09e5413130b6e86b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:10.235318 30392 raft_consensus.cc:729] T e2ee001bc1f24291b07428d419ca6141 P 6b6ff915e24e4cf09e5413130b6e86b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b6ff915e24e4cf09e5413130b6e86b3, State: Initialized, Role: FOLLOWER
I20260807 22:48:10.235467 30392 consensus_queue.cc:260] T e2ee001bc1f24291b07428d419ca6141 P 6b6ff915e24e4cf09e5413130b6e86b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e560779109c34f28923d6758135495ac" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42625 } } peers { permanent_uuid: "6bcbc13c64c9462bb899341481f34769" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44067 } } peers { permanent_uuid: "6b6ff915e24e4cf09e5413130b6e86b3" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39989 } }
I20260807 22:48:10.235818 30392 ts_tablet_manager.cc:1428] T e2ee001bc1f24291b07428d419ca6141 P 6b6ff915e24e4cf09e5413130b6e86b3: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260807 22:48:10.235999 30082 heartbeater.cc:503] Master 127.16.98.190:34675 was elected leader, sending a full tablet report...
W20260807 22:48:10.375294 30357 tablet.cc:2369] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:48:10.406818 30220 tablet.cc:2369] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:48:10.423341 30396 raft_consensus.cc:482] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:48:10.423511 30396 raft_consensus.cc:504] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e560779109c34f28923d6758135495ac" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42625 } } peers { permanent_uuid: "6bcbc13c64c9462bb899341481f34769" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44067 } } peers { permanent_uuid: "6b6ff915e24e4cf09e5413130b6e86b3" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39989 } }
I20260807 22:48:10.423907 30396 leader_election.cc:288] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bcbc13c64c9462bb899341481f34769 (127.16.98.130:44067), 6b6ff915e24e4cf09e5413130b6e86b3 (127.16.98.129:39989)
I20260807 22:48:10.428910 30387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:10.424856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:49093 (local address 127.16.98.129:39989)
0807 22:48:10.424901 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:10.424903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:10.425227 (+ 324us) server_negotiation.cc:407] Connection header received
0807 22:48:10.425325 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:10.425326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:10.425335 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:10.425364 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:10.425595 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.426318 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.427847 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.428426 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.428480 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:10.428490 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:10.428786 (+ 296us) server_negotiation.cc:299] Negotiation successful
0807 22:48:10.428831 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 22:48:10.429404 30400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:10.424619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:39989 (local address 127.16.98.131:49093)
0807 22:48:10.425172 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:10.425182 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:10.425252 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:10.425431 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:10.425432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:10.425433 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:10.425515 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:10.425517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.426447 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.426448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:10.427739 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:10.427742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.428621 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.428622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:10.428699 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:10.428703 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:48:10.429261 (+ 558us) client_negotiation.cc:240] Negotiation successful
0807 22:48:10.429308 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":68,"threads_started":1}
I20260807 22:48:10.429873 30036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ee001bc1f24291b07428d419ca6141" candidate_uuid: "e560779109c34f28923d6758135495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b6ff915e24e4cf09e5413130b6e86b3" is_pre_election: true
I20260807 22:48:10.430053 30036 raft_consensus.cc:2454] T e2ee001bc1f24291b07428d419ca6141 P 6b6ff915e24e4cf09e5413130b6e86b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e560779109c34f28923d6758135495ac in term 0.
I20260807 22:48:10.430233 30389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:10.424958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:33003 (local address 127.16.98.130:44067)
0807 22:48:10.425012 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:10.425014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:10.425023 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:48:10.425053 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:10.425054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:10.425064 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:10.425131 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:10.425385 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.426128 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.429226 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.429806 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.429862 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:10.429871 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:10.430118 (+ 247us) server_negotiation.cc:299] Negotiation successful
0807 22:48:10.430164 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260807 22:48:10.430459 30242 leader_election.cc:302] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b6ff915e24e4cf09e5413130b6e86b3, e560779109c34f28923d6758135495ac; no voters:
I20260807 22:48:10.430562 30399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:10.424357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:44067 (local address 127.16.98.131:33003)
0807 22:48:10.424713 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:10.424730 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:10.424811 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:10.425171 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:10.425172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:10.425174 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:10.425272 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:10.425274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.427899 (+ 2625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.427900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:10.429099 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:10.429103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.429938 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.429939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:10.430026 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:10.430031 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:48:10.430373 (+ 342us) client_negotiation.cc:240] Negotiation successful
0807 22:48:10.430419 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":120,"threads_started":1}
I20260807 22:48:10.430647 30396 raft_consensus.cc:2781] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:48:10.430703 30396 raft_consensus.cc:482] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:48:10.430727 30396 raft_consensus.cc:3037] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:10.430817 30171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ee001bc1f24291b07428d419ca6141" candidate_uuid: "e560779109c34f28923d6758135495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bcbc13c64c9462bb899341481f34769" is_pre_election: true
I20260807 22:48:10.430955 30171 raft_consensus.cc:2454] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e560779109c34f28923d6758135495ac in term 0.
I20260807 22:48:10.431824 30396 raft_consensus.cc:504] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e560779109c34f28923d6758135495ac" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42625 } } peers { permanent_uuid: "6bcbc13c64c9462bb899341481f34769" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44067 } } peers { permanent_uuid: "6b6ff915e24e4cf09e5413130b6e86b3" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39989 } }
I20260807 22:48:10.431977 30396 leader_election.cc:288] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [CANDIDATE]: Term 1 election: Requested vote from peers 6bcbc13c64c9462bb899341481f34769 (127.16.98.130:44067), 6b6ff915e24e4cf09e5413130b6e86b3 (127.16.98.129:39989)
I20260807 22:48:10.432168 30171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ee001bc1f24291b07428d419ca6141" candidate_uuid: "e560779109c34f28923d6758135495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bcbc13c64c9462bb899341481f34769"
I20260807 22:48:10.432165 30036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ee001bc1f24291b07428d419ca6141" candidate_uuid: "e560779109c34f28923d6758135495ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b6ff915e24e4cf09e5413130b6e86b3"
I20260807 22:48:10.432248 30036 raft_consensus.cc:3037] T e2ee001bc1f24291b07428d419ca6141 P 6b6ff915e24e4cf09e5413130b6e86b3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:10.432250 30171 raft_consensus.cc:3037] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:10.433256 30171 raft_consensus.cc:2454] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e560779109c34f28923d6758135495ac in term 1.
I20260807 22:48:10.433272 30036 raft_consensus.cc:2454] T e2ee001bc1f24291b07428d419ca6141 P 6b6ff915e24e4cf09e5413130b6e86b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e560779109c34f28923d6758135495ac in term 1.
I20260807 22:48:10.433501 30242 leader_election.cc:302] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bcbc13c64c9462bb899341481f34769, e560779109c34f28923d6758135495ac; no voters:
I20260807 22:48:10.433622 30396 raft_consensus.cc:2781] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:48:10.433789 30396 raft_consensus.cc:686] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [term 1 LEADER]: Becoming Leader. State: Replica: e560779109c34f28923d6758135495ac, State: Running, Role: LEADER
I20260807 22:48:10.433885 30396 consensus_queue.cc:237] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e560779109c34f28923d6758135495ac" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42625 } } peers { permanent_uuid: "6bcbc13c64c9462bb899341481f34769" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44067 } } peers { permanent_uuid: "6b6ff915e24e4cf09e5413130b6e86b3" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39989 } }
I20260807 22:48:10.434743 29878 catalog_manager.cc:5495] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac reported cstate change: term changed from 0 to 1, leader changed from <none> to e560779109c34f28923d6758135495ac (127.16.98.131). New cstate: current_term: 1 leader_uuid: "e560779109c34f28923d6758135495ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e560779109c34f28923d6758135495ac" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bcbc13c64c9462bb899341481f34769" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b6ff915e24e4cf09e5413130b6e86b3" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39989 } health_report { overall_health: UNKNOWN } } }
W20260807 22:48:10.458343 30083 tablet.cc:2369] T e2ee001bc1f24291b07428d419ca6141 P 6b6ff915e24e4cf09e5413130b6e86b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:48:10.476648 30388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:10.473118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.16.98.131:42625)
0807 22:48:10.473161 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:10.473163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:10.473170 (+ 7us) server_negotiation.cc:407] Connection header received
0807 22:48:10.473212 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:10.473213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:10.473234 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:10.473295 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:48:10.473458 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.474149 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.475428 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:10.475734 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:10.475810 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:10.476052 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:48:10.476204 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:48:10.476283 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:10.476502 (+ 219us) server_negotiation.cc:299] Negotiation successful
0807 22:48:10.476527 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 22:48:10.479460 30036 raft_consensus.cc:1261] T e2ee001bc1f24291b07428d419ca6141 P 6b6ff915e24e4cf09e5413130b6e86b3 [term 1 FOLLOWER]: Refusing update from remote peer e560779109c34f28923d6758135495ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:48:10.479835 30171 raft_consensus.cc:1261] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769 [term 1 FOLLOWER]: Refusing update from remote peer e560779109c34f28923d6758135495ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:48:10.480254 30396 consensus_queue.cc:1035] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b6ff915e24e4cf09e5413130b6e86b3" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:48:10.480448 30396 consensus_queue.cc:1035] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bcbc13c64c9462bb899341481f34769" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:48:10.484665 30406 mvcc.cc:205] Tried to move back new op lower bound from 7316041279400341504 to 7316041279217946624. Current Snapshot: MvccSnapshot[applied={T|T < 7316041279400341504}]
I20260807 22:48:10.489020 29877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:48:10.489140 29877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:48:10.489970 29877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 537c05c1b5d1475dbbad7470f20e4a18: Sending DeleteTablet for 3 replicas of tablet e2ee001bc1f24291b07428d419ca6141
I20260807 22:48:10.490554 30152 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ee001bc1f24291b07428d419ca6141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:10 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:60852
I20260807 22:48:10.490849 30015 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ee001bc1f24291b07428d419ca6141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:10 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:47706
I20260807 22:48:10.491159 30290 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ee001bc1f24291b07428d419ca6141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:10 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:49474
I20260807 22:48:10.491863 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 29918
I20260807 22:48:10.495826 30407 mvcc.cc:205] Tried to move back new op lower bound from 7316041279400341504 to 7316041279217946624. Current Snapshot: MvccSnapshot[applied={T|T < 7316041279400341504 or (T in {7316041279400341504})}]
I20260807 22:48:10.500743 30414 tablet_replica.cc:330] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769: stopping tablet replica
I20260807 22:48:10.500918 30414 raft_consensus.cc:2229] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:48:10.502302 30414 raft_consensus.cc:2258] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:48:10.502511 30416 tablet_replica.cc:330] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac: stopping tablet replica
I20260807 22:48:10.502871 30416 raft_consensus.cc:2229] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:48:10.503002 30416 raft_consensus.cc:2258] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 07 22:48:10 dist-test-slave-rgb1 krb5kdc[28861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for HTTP/127.16.98.148@KRBTEST.COM
I20260807 22:48:10.508090 30416 ts_tablet_manager.cc:1905] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:48:10.506152 30414 ts_tablet_manager.cc:1905] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:48:10.539728 30414 ts_tablet_manager.cc:1918] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:48:10.539846 30414 log.cc:1196] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/e2ee001bc1f24291b07428d419ca6141
I20260807 22:48:10.540163 30414 ts_tablet_manager.cc:1939] T e2ee001bc1f24291b07428d419ca6141 P 6bcbc13c64c9462bb899341481f34769: Deleting consensus metadata
I20260807 22:48:10.542577 30416 ts_tablet_manager.cc:1918] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:48:10.542680 30416 log.cc:1196] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/e2ee001bc1f24291b07428d419ca6141
I20260807 22:48:10.542997 30416 ts_tablet_manager.cc:1939] T e2ee001bc1f24291b07428d419ca6141 P e560779109c34f28923d6758135495ac: Deleting consensus metadata
W20260807 22:48:10.544739 29863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:39989 (error 108)
I20260807 22:48:10.545332 29863 catalog_manager.cc:4841] TS 6bcbc13c64c9462bb899341481f34769 (127.16.98.130:44067): tablet e2ee001bc1f24291b07428d419ca6141 (table test-table [id=9d456c56736740a38c885e6ad0d457c5]) successfully deleted
I20260807 22:48:10.545580 29865 catalog_manager.cc:4841] TS e560779109c34f28923d6758135495ac (127.16.98.131:42625): tablet e2ee001bc1f24291b07428d419ca6141 (table test-table [id=9d456c56736740a38c885e6ad0d457c5]) successfully deleted
I20260807 22:48:10.545773 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 30087
I20260807 22:48:10.553196 30385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:10.552991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:39989 (local address 127.0.0.1:47712)
0807 22:48:10.553076 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:10.553144 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260807 22:48:10.553380 29863 catalog_manager.cc:4584] TS 6b6ff915e24e4cf09e5413130b6e86b3 (127.16.98.129:39989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2ee001bc1f24291b07428d419ca6141: Network error: Client connection negotiation failed: client connection to 127.16.98.129:39989: connect: Connection refused (error 111)
I20260807 22:48:10.564807 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 30222
I20260807 22:48:10.576050 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 29845
I20260807 22:48:10.586297 16778 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 22:48:10.932271 16778 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 22:48:10.932422 16778 mini_ranger.cc:66] Stopping Ranger...
I20260807 22:48:11.288064 16778 mini_ranger.cc:68] Stopped Ranger
2026-08-07 22:48:11.288 UTC [28891] LOG: received smart shutdown request
2026-08-07 22:48:11.293 UTC [28891] LOG: background worker "logical replication launcher" (PID 28901) exited with exit code 1
2026-08-07 22:48:11.294 UTC [28895] LOG: shutting down
2026-08-07 22:48:11.544 UTC [28891] LOG: database system is shut down
2026-08-07T22:48:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58241 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:48:11 dist-test-slave-rgb1 krb5kdc[30427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:48:11 dist-test-slave-rgb1 krb5kdc[30427](info): set up 2 sockets
Aug 07 22:48:11 dist-test-slave-rgb1 krb5kdc[30427](info): commencing operation
krb5kdc: starting...
W20260807 22:48:12.880935 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:48:12 dist-test-slave-rgb1 krb5kdc[30427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:48:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:48:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:48:13.041524 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:46597
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:46597,127.16.98.189:40979,127.16.98.188:39781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:13.143700 30440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:13.143962 30440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:13.143989 30440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:13.145275 30440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:48:13.145332 30440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:13.145347 30440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:13.145360 30440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:48:13.145373 30440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:48:13.147037 30440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:46597,127.16.98.189:40979,127.16.98.188:39781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:46597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:13.147346 30440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:13.147627 30440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:13.154422 30449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:13.154422 30446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:13.154445 30447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:13.154479 30440 server_base.cc:1034] running on GCE node
I20260807 22:48:13.155131 30440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:13.155457 30440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:13.156652 30440 hybrid_clock.cc:648] HybridClock initialized: now 1786142893156626 us; error 44 us; skew 500 ppm
Aug 07 22:48:13 dist-test-slave-rgb1 krb5kdc[30427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:13.160512 30440 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:48:13.161271 30440 webserver.cc:460] Webserver started at http://127.16.98.190:37429/ using document root <none> and password file <none>
I20260807 22:48:13.161559 30440 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:13.161641 30440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:13.161832 30440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:48:13.163842 30440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "6cc58513e85c45d08658534345b25c91"
format_stamp: "Formatted at 2026-08-07 22:48:13 on dist-test-slave-rgb1"
server_key: "a01e377bb5b20ee399ea915dd2ce1d2c"
server_key_iv: "cb01880de9a2598440063cdabe20f177"
server_key_version: "encryptionkey@0"
I20260807 22:48:13.164770 30440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "6cc58513e85c45d08658534345b25c91"
format_stamp: "Formatted at 2026-08-07 22:48:13 on dist-test-slave-rgb1"
server_key: "a01e377bb5b20ee399ea915dd2ce1d2c"
server_key_iv: "cb01880de9a2598440063cdabe20f177"
server_key_version: "encryptionkey@0"
I20260807 22:48:13.166683 30440 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260807 22:48:13.167855 30456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:13.168186 30440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:13.168332 30440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "6cc58513e85c45d08658534345b25c91"
format_stamp: "Formatted at 2026-08-07 22:48:13 on dist-test-slave-rgb1"
server_key: "a01e377bb5b20ee399ea915dd2ce1d2c"
server_key_iv: "cb01880de9a2598440063cdabe20f177"
server_key_version: "encryptionkey@0"
I20260807 22:48:13.168435 30440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:13.194571 30440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:13.198107 30440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:13.198334 30440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:13.204612 30440 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:46597
I20260807 22:48:13.204636 30509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:46597 every 8 connection(s)
I20260807 22:48:13.205129 30440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:48:13.206195 30510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:13.207973 30510 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:48:13.209507 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 30440
I20260807 22:48:13.209671 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:48:13.209985 16778 external_mini_cluster.cc:1428] Setting key 8a341d519f9824c9b3c0bb77f8e43706
Aug 07 22:48:13 dist-test-slave-rgb1 krb5kdc[30427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.98.189@KRBTEST.COM; defaulting to no policy
I20260807 22:48:13.223387 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.209178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56180 (local address 127.16.98.190:46597)
0807 22:48:13.209502 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:13.209508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:13.209558 (+ 50us) server_negotiation.cc:407] Connection header received
0807 22:48:13.210136 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:13.210139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:13.210159 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:13.210261 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:13.210918 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.212237 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.213348 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.213898 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.214052 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.218254 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:13.218263 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:13.218264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:13.218272 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:13.220684 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.221509 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.221514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.221515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.221647 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.222229 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.222231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.222231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.222530 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:13.222700 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:13.222898 (+ 198us) server_negotiation.cc:299] Negotiation successful
0807 22:48:13.223109 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":288,"thread_start_us":233,"threads_started":1}
I20260807 22:48:13.223390 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.208863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:46597 (local address 127.0.0.1:56180)
0807 22:48:13.209131 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.209185 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:13.209943 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:13.210282 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:13.210285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:13.210620 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:13.210792 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.210797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.212350 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.212352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.213231 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.213233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.214099 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.214101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.214225 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.215321 (+ 1096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:13.215337 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:13.218056 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:13.220917 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.220933 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.220936 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.221338 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.221741 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.221742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.221742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.221997 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.222671 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:13.222678 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:13.222758 (+ 80us) client_negotiation.cc:769] Sending connection context
0807 22:48:13.222871 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 22:48:13.223163 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":136,"thread_start_us":177,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.189@KRBTEST.COM" created.
I20260807 22:48:13.224653 30510 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:48:13.225188 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.225025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40979 (local address 127.0.0.1:57980)
0807 22:48:13.225087 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.225148 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260807 22:48:13.225485 30457 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40979: connect: Connection refused (error 111)
I20260807 22:48:13.226150 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.226020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40979 (local address 127.0.0.1:57984)
0807 22:48:13.226077 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.226114 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260807 22:48:13.226341 30510 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.189:40979: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40979: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.189.keytab.
Entry for principal kudu/127.16.98.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.189.keytab.
I20260807 22:48:13.249257 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.189:40979
--webserver_interface=127.16.98.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.189.keytab
--principal=kudu/127.16.98.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:46597,127.16.98.189:40979,127.16.98.188:39781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:48:13.275678 30510 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } attempt: 1
I20260807 22:48:13.276261 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.276116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40979 (local address 127.0.0.1:57998)
0807 22:48:13.276189 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.276231 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260807 22:48:13.276907 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.276640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40979 (local address 127.0.0.1:58014)
0807 22:48:13.276785 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.276858 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260807 22:48:13.277125 30510 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.189:40979: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40979: connect: Connection refused (error 111)
I20260807 22:48:13.345320 30510 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } attempt: 2
I20260807 22:48:13.346061 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.345881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40979 (local address 127.0.0.1:58028)
0807 22:48:13.345962 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.346016 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 22:48:13.346741 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.346598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40979 (local address 127.0.0.1:58044)
0807 22:48:13.346668 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.346711 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 22:48:13.346961 30510 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.189:40979: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40979: connect: Connection refused (error 111)
W20260807 22:48:13.353886 30517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:13.354218 30517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:13.354246 30517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:13.355777 30517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:48:13.355842 30517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:13.355858 30517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:13.355875 30517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:48:13.355890 30517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:48:13.357932 30517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:46597,127.16.98.189:40979,127.16.98.188:39781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.189.keytab
--principal=kudu/127.16.98.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.189:40979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.98.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:13.358233 30517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:13.358521 30517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:13.364462 30523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:13.364487 30525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:13.364503 30522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:13.364717 30517 server_base.cc:1034] running on GCE node
I20260807 22:48:13.365155 30517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:13.365545 30517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:13.366750 30517 hybrid_clock.cc:648] HybridClock initialized: now 1786142893366718 us; error 48 us; skew 500 ppm
Aug 07 22:48:13 dist-test-slave-rgb1 krb5kdc[30427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:13.371344 30517 init.cc:377] Logged in from keytab as kudu/127.16.98.189@KRBTEST.COM (short username kudu)
I20260807 22:48:13.372368 30517 webserver.cc:460] Webserver started at http://127.16.98.189:46529/ using document root <none> and password file <none>
I20260807 22:48:13.372741 30517 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:13.372810 30517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:13.373009 30517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:48:13.374336 30517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/data/instance:
uuid: "9e71f9efe3654d57a9c746631f09287a"
format_stamp: "Formatted at 2026-08-07 22:48:13 on dist-test-slave-rgb1"
server_key: "d76aa81c46de36ed223eb71ae4974228"
server_key_iv: "6b4ed072ef521184286fd2cb34b78c9e"
server_key_version: "encryptionkey@0"
I20260807 22:48:13.374778 30517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/wal/instance:
uuid: "9e71f9efe3654d57a9c746631f09287a"
format_stamp: "Formatted at 2026-08-07 22:48:13 on dist-test-slave-rgb1"
server_key: "d76aa81c46de36ed223eb71ae4974228"
server_key_iv: "6b4ed072ef521184286fd2cb34b78c9e"
server_key_version: "encryptionkey@0"
I20260807 22:48:13.376698 30517 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 22:48:13.378609 30532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:13.378978 30517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:48:13.379122 30517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/wal
uuid: "9e71f9efe3654d57a9c746631f09287a"
format_stamp: "Formatted at 2026-08-07 22:48:13 on dist-test-slave-rgb1"
server_key: "d76aa81c46de36ed223eb71ae4974228"
server_key_iv: "6b4ed072ef521184286fd2cb34b78c9e"
server_key_version: "encryptionkey@0"
I20260807 22:48:13.379221 30517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:13.392455 30517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:13.395922 30517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:13.396137 30517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:13.403066 30517 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.189:40979
I20260807 22:48:13.403168 30587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.189:40979 every 8 connection(s)
I20260807 22:48:13.403654 30517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/data/info.pb
I20260807 22:48:13.405419 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:13.407209 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 30517
I20260807 22:48:13.407472 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-1/wal/instance
I20260807 22:48:13.407795 16778 external_mini_cluster.cc:1428] Setting key fd4082366cf41cc708149d30cebd6802
I20260807 22:48:13.408982 30588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:48:13 dist-test-slave-rgb1 krb5kdc[30427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.189@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.98.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.188@KRBTEST.COM" created.
I20260807 22:48:13.426476 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.410036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.16.98.190:46597)
0807 22:48:13.410101 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:13.410104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:13.410813 (+ 709us) server_negotiation.cc:407] Connection header received
0807 22:48:13.411710 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:13.411712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:13.411729 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:13.411963 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:13.412889 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.413454 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.415116 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.415640 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.415815 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.420500 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:13.420510 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:13.420510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:13.420532 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:13.422771 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.423917 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.423921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.423921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.424028 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.424908 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.424909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.424910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.425466 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:13.425837 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:13.426239 (+ 402us) server_negotiation.cc:299] Negotiation successful
0807 22:48:13.426396 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260807 22:48:13.426934 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.410266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:46597 (local address 127.0.0.1:56194)
0807 22:48:13.410585 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.410651 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:13.411535 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:13.412067 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:13.412072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:13.412450 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:13.412766 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.412773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.413569 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.413571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.414779 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.414787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.415849 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.415851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.416060 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.417665 (+ 1605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:13.417679 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:13.420035 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:13.422965 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.422987 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.422991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.423756 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.424182 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.424184 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.424184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.424346 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.425844 (+ 1498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:13.425854 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:13.426048 (+ 194us) client_negotiation.cc:769] Sending connection context
0807 22:48:13.426162 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 22:48:13.426402 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":140,"threads_started":1}
I20260807 22:48:13.428212 30588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:48:13 dist-test-slave-rgb1 krb5kdc[30427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.189@KRBTEST.COM for kudu/127.16.98.189@KRBTEST.COM
I20260807 22:48:13.438398 30510 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } attempt: 3
I20260807 22:48:13.439020 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.428782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40979 (local address 127.0.0.1:58056)
0807 22:48:13.428973 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.428987 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:13.429048 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:13.429434 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:13.429435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:13.429621 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:13.429680 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.429682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.430713 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.430714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.431236 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.431238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.431934 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.431935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.432026 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.432741 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:13.432748 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:13.434711 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:13.437691 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.437692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.437693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.437885 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.438165 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.438166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.438166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.438252 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.438608 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:13.438609 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:13.438657 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 22:48:13.438759 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 22:48:13.438893 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260807 22:48:13.439149 30593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.428978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.16.98.189:40979)
0807 22:48:13.429254 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:13.429259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:13.429282 (+ 23us) server_negotiation.cc:407] Connection header received
0807 22:48:13.429305 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:13.429306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:13.429320 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:13.429442 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:13.429774 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.430605 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.431325 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.431649 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.431924 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.434826 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:13.434839 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:13.434839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:13.434847 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:13.437551 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.438005 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.438008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.438009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.438078 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.438329 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.438330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.438330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.438524 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:13.438634 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:13.438814 (+ 180us) server_negotiation.cc:299] Negotiation successful
0807 22:48:13.438990 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":242,"thread_start_us":163,"threads_started":1}
I20260807 22:48:13.440196 30588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:48:13.441282 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.441066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:39781 (local address 127.0.0.1:36380)
0807 22:48:13.441169 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.441246 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260807 22:48:13.441606 30536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
I20260807 22:48:13.442441 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.442255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:39781 (local address 127.0.0.1:36384)
0807 22:48:13.442354 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.442405 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260807 22:48:13.442927 30588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:39781: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
Aug 07 22:48:13 dist-test-slave-rgb1 krb5kdc[30427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for kudu/127.16.98.189@KRBTEST.COM
I20260807 22:48:13.450649 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.439290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40979 (local address 127.0.0.1:58066)
0807 22:48:13.439401 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.439414 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:13.439454 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:13.439903 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:13.439904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:13.440127 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:13.440305 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.440308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.441905 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.441907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.442817 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.442820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.443520 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.443523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.443656 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.444906 (+ 1250us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:13.444912 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:13.447035 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:13.449288 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.449290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.449290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.449526 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.449872 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.449873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.449873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.449949 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.450236 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:13.450237 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:13.450334 (+ 97us) client_negotiation.cc:769] Sending connection context
0807 22:48:13.450443 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 22:48:13.450567 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 22:48:13.450766 30593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.439448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58066 (local address 127.16.98.189:40979)
0807 22:48:13.439601 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:13.439604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:13.439613 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:48:13.439634 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:13.439634 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:13.439646 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:13.439748 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:13.440473 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.441774 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.442934 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.443276 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.443659 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.447303 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:13.447305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:13.447306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:13.447316 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:13.449168 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.449644 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.449646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.449646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.449712 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.450012 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.450013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.450013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.450152 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:13.450248 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:13.450497 (+ 249us) server_negotiation.cc:299] Negotiation successful
0807 22:48:13.450627 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":112,"threads_started":1}
I20260807 22:48:13.451602 30510 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:48:13.452984 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.452805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:39781 (local address 127.0.0.1:36400)
0807 22:48:13.452894 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.452944 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.188.keytab.
Entry for principal kudu/127.16.98.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.188.keytab.
I20260807 22:48:13.453620 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.453481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:39781 (local address 127.0.0.1:36406)
0807 22:48:13.453566 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.453598 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260807 22:48:13.453860 30510 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:39781: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
I20260807 22:48:13.454252 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.188:39781
--webserver_interface=127.16.98.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.188.keytab
--principal=kudu/127.16.98.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:46597,127.16.98.189:40979,127.16.98.188:39781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:48:13.485145 30510 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } attempt: 1
I20260807 22:48:13.486136 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.485863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:39781 (local address 127.0.0.1:36410)
0807 22:48:13.485991 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.486049 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260807 22:48:13.486927 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.486649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:39781 (local address 127.0.0.1:36422)
0807 22:48:13.486779 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.486890 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260807 22:48:13.487185 30510 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:39781: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
I20260807 22:48:13.492292 30588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } attempt: 1
I20260807 22:48:13.493232 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.492813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:39781 (local address 127.0.0.1:36426)
0807 22:48:13.493138 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.493196 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295}
I20260807 22:48:13.493872 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.493616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:39781 (local address 127.0.0.1:36440)
0807 22:48:13.493751 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.493826 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260807 22:48:13.494078 30588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:39781: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
I20260807 22:48:13.562376 30588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } attempt: 2
I20260807 22:48:13.562461 30510 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } attempt: 2
I20260807 22:48:13.563005 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.562828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:39781 (local address 127.0.0.1:36442)
0807 22:48:13.562896 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.562965 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260807 22:48:13.563089 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.562910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:39781 (local address 127.0.0.1:36448)
0807 22:48:13.563002 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.563047 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 22:48:13.563614 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.563479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:39781 (local address 127.0.0.1:36470)
0807 22:48:13.563536 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.563573 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260807 22:48:13.563737 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.563312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:39781 (local address 127.0.0.1:36454)
0807 22:48:13.563548 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.563579 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":167,"threads_started":1}
W20260807 22:48:13.563931 30588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:39781: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
W20260807 22:48:13.563957 30510 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:39781: Network error: Client connection negotiation failed: client connection to 127.16.98.188:39781: connect: Connection refused (error 111)
W20260807 22:48:13.571981 30596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:13.572208 30596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:13.572265 30596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:13.574537 30596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:48:13.574628 30596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:13.574657 30596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:13.574678 30596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:48:13.574707 30596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:48:13.576849 30596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:46597,127.16.98.189:40979,127.16.98.188:39781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.188.keytab
--principal=kudu/127.16.98.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.188:39781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.98.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:13.577175 30596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:13.577442 30596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:13.585441 30604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:13.585453 30606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:13.585500 30603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:13.586782 30596 server_base.cc:1034] running on GCE node
I20260807 22:48:13.587158 30596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:13.587785 30596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:13.589214 30596 hybrid_clock.cc:648] HybridClock initialized: now 1786142893589169 us; error 76 us; skew 500 ppm
Aug 07 22:48:13 dist-test-slave-rgb1 krb5kdc[30427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:13.593422 30596 init.cc:377] Logged in from keytab as kudu/127.16.98.188@KRBTEST.COM (short username kudu)
I20260807 22:48:13.594136 30596 webserver.cc:460] Webserver started at http://127.16.98.188:34707/ using document root <none> and password file <none>
I20260807 22:48:13.594422 30596 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:13.594511 30596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:13.594686 30596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:48:13.596345 30596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/data/instance:
uuid: "44f29a09a4f74e7bb3a420b819342612"
format_stamp: "Formatted at 2026-08-07 22:48:13 on dist-test-slave-rgb1"
server_key: "9cd4e14609d1f8f9a62a44ec3484b0f8"
server_key_iv: "378ddb8372d7d7b8f76d58809861592f"
server_key_version: "encryptionkey@0"
I20260807 22:48:13.596884 30596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/wal/instance:
uuid: "44f29a09a4f74e7bb3a420b819342612"
format_stamp: "Formatted at 2026-08-07 22:48:13 on dist-test-slave-rgb1"
server_key: "9cd4e14609d1f8f9a62a44ec3484b0f8"
server_key_iv: "378ddb8372d7d7b8f76d58809861592f"
server_key_version: "encryptionkey@0"
I20260807 22:48:13.598858 30596 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 22:48:13.599972 30613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:13.600485 30596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:13.600641 30596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/wal
uuid: "44f29a09a4f74e7bb3a420b819342612"
format_stamp: "Formatted at 2026-08-07 22:48:13 on dist-test-slave-rgb1"
server_key: "9cd4e14609d1f8f9a62a44ec3484b0f8"
server_key_iv: "378ddb8372d7d7b8f76d58809861592f"
server_key_version: "encryptionkey@0"
I20260807 22:48:13.600740 30596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:13.638592 30596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:13.641160 30596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:13.641296 30596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:13.645884 30596 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.188:39781
I20260807 22:48:13.645924 30665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.188:39781 every 8 connection(s)
I20260807 22:48:13.646302 30596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/data/info.pb
I20260807 22:48:13.647423 30666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:13.649333 30666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:48:13.654333 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 30596
I20260807 22:48:13.654518 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786142716408518-16778-0/minicluster-data/master-2/wal/instance
I20260807 22:48:13.654778 16778 external_mini_cluster.cc:1428] Setting key b6fecb6c23fbd2d38c006ec61eae9ad2
I20260807 22:48:13.655102 30588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } attempt: 3
Aug 07 22:48:13 dist-test-slave-rgb1 krb5kdc[30427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.188@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
Aug 07 22:48:13 dist-test-slave-rgb1 krb5kdc[30427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.189@KRBTEST.COM for kudu/127.16.98.188@KRBTEST.COM
Aug 07 22:48:13 dist-test-slave-rgb1 krb5kdc[30427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.188@KRBTEST.COM
I20260807 22:48:13.661789 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.650230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56202 (local address 127.16.98.190:46597)
0807 22:48:13.650314 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:13.650317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:13.650670 (+ 353us) server_negotiation.cc:407] Connection header received
0807 22:48:13.651336 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:13.651337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:13.651359 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:13.651432 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:13.652237 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.652909 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.654104 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.654310 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.654377 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.657781 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:13.657782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:13.657783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:13.657792 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:13.659799 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.660373 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.660375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.660375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.660452 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.660849 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.660849 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.660850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.661035 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:13.661140 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:13.661551 (+ 411us) server_negotiation.cc:299] Negotiation successful
0807 22:48:13.661669 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 22:48:13.662326 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.650213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:46597 (local address 127.0.0.1:56202)
0807 22:48:13.650517 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.650559 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:13.651211 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:13.651571 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:13.651575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:13.651873 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:13.652121 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.652126 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.653089 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.653090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.654007 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.654010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.654415 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.654417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.654616 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.655548 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:13.655560 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:13.657676 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:13.659905 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.659914 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.659916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.660266 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.660549 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.660550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.660551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.660733 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.661136 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:13.661145 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:13.661343 (+ 198us) client_negotiation.cc:769] Sending connection context
0807 22:48:13.661942 (+ 599us) client_negotiation.cc:240] Negotiation successful
0807 22:48:13.662140 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":147,"threads_started":1}
I20260807 22:48:13.663120 30510 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } attempt: 3
I20260807 22:48:13.663401 30666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:48:13.665966 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.655401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:39781 (local address 127.0.0.1:36494)
0807 22:48:13.655538 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.655554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:13.655612 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:13.656082 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:13.656083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:13.656272 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:13.656337 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.656339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.657610 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.657612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.658511 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.658514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.659240 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.659242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.659314 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.659768 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:13.659775 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:13.661401 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:13.664633 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.664634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.664635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.664881 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.665144 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.665145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.665146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.665218 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.665612 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:13.665613 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:13.665661 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 22:48:13.665736 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 22:48:13.665843 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260807 22:48:13.666397 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.655779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.16.98.188:39781)
0807 22:48:13.655926 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:13.655928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:13.655936 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:48:13.655953 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:13.655954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:13.655962 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:13.656056 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:13.656431 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.657496 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.658611 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.659144 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.659223 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.661504 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:13.661513 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:13.661514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:13.661522 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:13.664508 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.664981 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.664985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.664985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.665056 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.665301 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.665302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.665303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.665522 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:13.666122 (+ 600us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:13.666176 (+ 54us) server_negotiation.cc:299] Negotiation successful
0807 22:48:13.666285 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260807 22:48:13.666760 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.655630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.16.98.188:39781)
0807 22:48:13.655836 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:13.655841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:13.655882 (+ 41us) server_negotiation.cc:407] Connection header received
0807 22:48:13.655915 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:13.655918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:13.655946 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:13.656146 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:13.657064 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.657810 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.658961 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.659277 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.659345 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.662606 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:13.662609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:13.662609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:13.662617 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:13.665159 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.665609 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.665610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.665611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.665678 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.665919 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.665920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.665920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.666062 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:13.666206 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:13.666553 (+ 347us) server_negotiation.cc:299] Negotiation successful
0807 22:48:13.666628 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":101,"threads_started":1}
Aug 07 22:48:13 dist-test-slave-rgb1 krb5kdc[30427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.188@KRBTEST.COM for kudu/127.16.98.189@KRBTEST.COM
Aug 07 22:48:13 dist-test-slave-rgb1 krb5kdc[30427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for kudu/127.16.98.188@KRBTEST.COM
I20260807 22:48:13.670259 30588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a: Bootstrap starting.
I20260807 22:48:13.672617 30588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:13.673204 30588 log.cc:824] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a: Log is configured to *not* fsync() on all Append() calls
Aug 07 22:48:13 dist-test-slave-rgb1 krb5kdc[30427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:13.674669 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.663990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40979 (local address 127.0.0.1:58074)
0807 22:48:13.664034 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.664045 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:13.664085 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:13.664222 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:13.664223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:13.664434 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:13.664473 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.664475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.665491 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.665492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.666789 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.666792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.667299 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.667300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.667369 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.668193 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:13.668200 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:13.670555 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:13.672917 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.672918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.672919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.673402 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.673683 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.673684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.673685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.673753 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.674305 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:13.674306 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:13.674361 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 22:48:13.674436 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 22:48:13.674550 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260807 22:48:13.674909 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.663414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:39781 (local address 127.0.0.1:36496)
0807 22:48:13.663515 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.663527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:13.663574 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:13.665451 (+ 1877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:13.665452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:13.665632 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:13.665685 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.665687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.666783 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.666784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.667649 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.667652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.668207 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.668208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.668271 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.668777 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:13.668784 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:13.670544 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:13.673505 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.673506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.673507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.673750 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.674001 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.674002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.674002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.674191 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.674551 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:13.674552 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:13.674607 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 22:48:13.674666 (+ 59us) client_negotiation.cc:240] Negotiation successful
0807 22:48:13.674777 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 22:48:13.675379 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.663678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.16.98.188:39781)
0807 22:48:13.665289 (+ 1611us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:13.665291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:13.665299 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:48:13.665319 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:13.665320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:13.665331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:13.665767 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:13.665805 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.666661 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.667757 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.668110 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.668436 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.670676 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:13.670677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:13.670677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:13.670684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:13.673325 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.673855 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.673856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.673857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.673925 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.674285 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.674286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.674287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.674453 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:13.675115 (+ 662us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:13.675166 (+ 51us) server_negotiation.cc:299] Negotiation successful
0807 22:48:13.675257 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1589,"thread_start_us":98,"threads_started":1}
I20260807 22:48:13.675434 30588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a: No bootstrap required, opened a new log
I20260807 22:48:13.675913 30593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.664034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.16.98.189:40979)
0807 22:48:13.664080 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:13.664083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:13.664092 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:48:13.664155 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:13.664156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:13.664166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:13.664221 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:13.664617 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.665390 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.666871 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.667174 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.667245 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.670688 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:13.670690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:13.670690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:13.670699 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:13.672797 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.673518 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.673519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.673519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.673588 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.673869 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.673870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.673871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.674181 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:13.674247 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:13.674478 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 22:48:13.674567 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 22:48:13.676342 30666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:48:13.677774 30510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91: Bootstrap starting.
I20260807 22:48:13.677780 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.667949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56218 (local address 127.16.98.190:46597)
0807 22:48:13.667998 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:13.668000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:13.668008 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:48:13.668030 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:13.668031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:13.668040 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:13.668105 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:13.668922 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.669681 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.670663 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.670956 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.670997 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.674358 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:13.674360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:13.674360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:13.674369 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:13.676325 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.676855 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.676856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.676856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.676930 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.677264 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.677265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.677265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.677384 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:13.677558 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:13.677607 (+ 49us) server_negotiation.cc:299] Negotiation successful
0807 22:48:13.677683 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 22:48:13.678877 30588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } }
I20260807 22:48:13.679086 30588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:13.679133 30588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e71f9efe3654d57a9c746631f09287a, State: Initialized, Role: FOLLOWER
I20260807 22:48:13.679153 30510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:13.679279 30588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } }
I20260807 22:48:13.679526 30510 log.cc:824] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:13.679721 30588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:48:13.679690 30674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } } }
I20260807 22:48:13.679837 30674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a [sys.catalog]: This master's current role is: FOLLOWER
W20260807 22:48:13.682050 30686 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:48:13.682119 30686 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 07 22:48:13 dist-test-slave-rgb1 krb5kdc[30427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.188@KRBTEST.COM for kudu/127.16.98.188@KRBTEST.COM
Aug 07 22:48:13 dist-test-slave-rgb1 krb5kdc[30427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.189@KRBTEST.COM
I20260807 22:48:13.684382 30517 master_runner.cc:186] Error getting master registration for 127.16.98.190:46597: OK, instance_id { permanent_uuid: "6cc58513e85c45d08658534345b25c91" instance_seqno: 1786142893203143 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 22:48:13.688709 30517 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 22:48:13.689057 30510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91: No bootstrap required, opened a new log
I20260807 22:48:13.689692 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.680223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:39781 (local address 127.0.0.1:36500)
0807 22:48:13.680322 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:13.680335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:13.680386 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:13.680759 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:13.680760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:13.680942 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:13.680998 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.681000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.681972 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.681973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.682753 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:13.682757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.683402 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.683403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:13.683451 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.683949 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:13.683955 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:13.686463 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:13.688390 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.688391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.688391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.688671 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.688978 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:13.688978 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:13.688978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:13.689029 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:13.689366 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:13.689367 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:13.689403 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 22:48:13.689463 (+ 60us) client_negotiation.cc:240] Negotiation successful
0807 22:48:13.689582 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 22:48:13.690192 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.680585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.16.98.188:39781)
0807 22:48:13.680626 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:13.680629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:13.680637 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:48:13.680660 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:13.680661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:13.680674 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:13.680739 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:13.681110 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.681885 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.682903 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.683219 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.683324 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.686582 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:13.686584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:13.686584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:13.686591 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:13.688291 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.688846 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.688847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.688847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.688917 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.689088 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.689088 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.689089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.689281 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:13.689344 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:13.690012 (+ 668us) server_negotiation.cc:299] Negotiation successful
0807 22:48:13.690110 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260807 22:48:13.690503 30593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.680490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.16.98.189:40979)
0807 22:48:13.680540 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:13.680541 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:13.680548 (+ 7us) server_negotiation.cc:407] Connection header received
0807 22:48:13.680601 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:13.680602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:13.680613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:13.680664 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:13.681319 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.682055 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.683141 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.683422 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.683484 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.686599 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:13.686600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:13.686601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:13.686611 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:13.688423 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.688979 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.688980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.688981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.689057 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.689318 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.689319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.689319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.689464 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:13.689613 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:13.690206 (+ 593us) server_negotiation.cc:299] Negotiation successful
0807 22:48:13.690395 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260807 22:48:13.691921 30666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612: Bootstrap starting.
I20260807 22:48:13.692162 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:48:13.692695 30666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:13.692983 30666 log.cc:824] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:13.694218 30510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } }
I20260807 22:48:13.694371 30510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:13.694419 30510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc58513e85c45d08658534345b25c91, State: Initialized, Role: FOLLOWER
I20260807 22:48:13.694674 30510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } }
I20260807 22:48:13.694999 30695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } } }
I20260807 22:48:13.695075 30510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:48:13.695092 30695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 22:48:13.696975 30706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:48:13.697050 30706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:48:13.696936 30666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612: No bootstrap required, opened a new log
I20260807 22:48:13.699224 30666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } }
I20260807 22:48:13.699364 30666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:13.699388 30666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f29a09a4f74e7bb3a420b819342612, State: Initialized, Role: FOLLOWER
W20260807 22:48:13.699625 30463 tablet.cc:2369] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:48:13.699680 30666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } }
I20260807 22:48:13.700089 30711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } } }
I20260807 22:48:13.700145 30666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:48:13.700230 30711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 22:48:13.702476 30722 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:48:13.702557 30722 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:48:13.703359 30440 master_runner.cc:263] No leader master found from master 6cc58513e85c45d08658534345b25c91
I20260807 22:48:13.703418 30440 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 22:48:13.705029 30596 master_runner.cc:263] No leader master found from master 44f29a09a4f74e7bb3a420b819342612
I20260807 22:48:13.705087 30596 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 22:48:13.722369 30711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:48:13.722514 30711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } }
I20260807 22:48:13.722797 30711 leader_election.cc:288] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc58513e85c45d08658534345b25c91 (127.16.98.190:46597), 9e71f9efe3654d57a9c746631f09287a (127.16.98.189:40979)
I20260807 22:48:13.723275 30482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f29a09a4f74e7bb3a420b819342612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc58513e85c45d08658534345b25c91" is_pre_election: true
I20260807 22:48:13.723464 30482 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f29a09a4f74e7bb3a420b819342612 in term 0.
I20260807 22:48:13.723668 30562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f29a09a4f74e7bb3a420b819342612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e71f9efe3654d57a9c746631f09287a" is_pre_election: true
I20260807 22:48:13.723852 30562 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f29a09a4f74e7bb3a420b819342612 in term 0.
I20260807 22:48:13.724085 30614 leader_election.cc:302] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f29a09a4f74e7bb3a420b819342612, 9e71f9efe3654d57a9c746631f09287a; no voters:
I20260807 22:48:13.724255 30711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:48:13.724314 30711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:48:13.724349 30711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:13.725417 30711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } }
I20260807 22:48:13.725593 30711 leader_election.cc:288] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc58513e85c45d08658534345b25c91 (127.16.98.190:46597), 9e71f9efe3654d57a9c746631f09287a (127.16.98.189:40979)
I20260807 22:48:13.726053 30562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f29a09a4f74e7bb3a420b819342612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e71f9efe3654d57a9c746631f09287a"
I20260807 22:48:13.726161 30562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:13.727214 30562 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f29a09a4f74e7bb3a420b819342612 in term 1.
I20260807 22:48:13.727423 30614 leader_election.cc:302] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f29a09a4f74e7bb3a420b819342612, 9e71f9efe3654d57a9c746631f09287a; no voters:
I20260807 22:48:13.727591 30482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f29a09a4f74e7bb3a420b819342612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc58513e85c45d08658534345b25c91"
I20260807 22:48:13.727698 30482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:13.728778 30482 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f29a09a4f74e7bb3a420b819342612 in term 1.
I20260807 22:48:13.729033 30711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:48:13.729259 30711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [term 1 LEADER]: Becoming Leader. State: Replica: 44f29a09a4f74e7bb3a420b819342612, State: Running, Role: LEADER
I20260807 22:48:13.729382 30711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } }
I20260807 22:48:13.729885 30711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f29a09a4f74e7bb3a420b819342612. Latest consensus state: current_term: 1 leader_uuid: "44f29a09a4f74e7bb3a420b819342612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } } }
I20260807 22:48:13.729985 30711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:13.732015 30593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.697564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58092 (local address 127.16.98.189:40979)
0807 22:48:13.697620 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:13.697622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:13.708784 (+ 11162us) server_negotiation.cc:407] Connection header received
0807 22:48:13.708881 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:13.708882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:13.708902 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:13.708975 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:13.710194 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.711040 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.712330 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.712757 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.712813 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.720094 (+ 7281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:13.720096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:13.720097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:13.720107 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:13.722014 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.722858 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.722860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.722860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.722926 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.731108 (+ 8182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.731110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.731110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.731278 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:13.731373 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:13.731786 (+ 413us) server_negotiation.cc:299] Negotiation successful
0807 22:48:13.731905 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260807 22:48:13.732981 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.697282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.16.98.190:46597)
0807 22:48:13.697320 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:13.697322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:13.703934 (+ 6612us) server_negotiation.cc:407] Connection header received
0807 22:48:13.704049 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:13.704051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:13.704063 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:13.704094 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:13.705350 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.706454 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.707675 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.708007 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.708059 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.717872 (+ 9813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:13.717874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:13.717875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:13.717886 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:13.719933 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.730244 (+ 10311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.730246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.730246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.730349 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.730775 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.730776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.730777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.730935 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:13.731036 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:13.732717 (+ 1681us) server_negotiation.cc:299] Negotiation successful
0807 22:48:13.732857 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260807 22:48:13.736729 30726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:48:13.737069 30726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:48:13.739136 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:13.696987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.16.98.188:39781)
0807 22:48:13.697037 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:13.697039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:13.709080 (+ 12041us) server_negotiation.cc:407] Connection header received
0807 22:48:13.709140 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:13.709142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:13.709155 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:13.709216 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:13.715701 (+ 6485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.716602 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.733171 (+ 16569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:13.733520 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:13.733577 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:13.735705 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:13.735708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:13.735708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:13.735719 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:13.737683 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.738159 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.738161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.738161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.738225 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:13.738438 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:13.738439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:13.738439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:13.738599 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:13.738694 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:13.738937 (+ 243us) server_negotiation.cc:299] Negotiation successful
0807 22:48:13.739033 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 22:48:13.741758 30482 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91 [term 1 FOLLOWER]: Refusing update from remote peer 44f29a09a4f74e7bb3a420b819342612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:48:13.742126 30711 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:48:13.743963 30695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f29a09a4f74e7bb3a420b819342612. Latest consensus state: current_term: 1 leader_uuid: "44f29a09a4f74e7bb3a420b819342612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } } }
I20260807 22:48:13.744102 30695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:48:13.744095 30562 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a [term 1 FOLLOWER]: Refusing update from remote peer 44f29a09a4f74e7bb3a420b819342612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:48:13.744413 30711 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:48:13.746169 30674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f29a09a4f74e7bb3a420b819342612. Latest consensus state: current_term: 1 leader_uuid: "44f29a09a4f74e7bb3a420b819342612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } } }
I20260807 22:48:13.746295 30674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:48:13.751113 30724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44f29a09a4f74e7bb3a420b819342612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } } }
I20260807 22:48:13.751245 30724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:13.751786 30695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44f29a09a4f74e7bb3a420b819342612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } } }
I20260807 22:48:13.751900 30695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:48:13.752033 30727 mvcc.cc:205] Tried to move back new op lower bound from 7316041292762877952 to 7316041292716384256. Current Snapshot: MvccSnapshot[applied={T|T < 7316041292762877952}]
I20260807 22:48:13.752138 30728 mvcc.cc:205] Tried to move back new op lower bound from 7316041292762877952 to 7316041292716384256. Current Snapshot: MvccSnapshot[applied={T|T < 7316041292762877952}]
I20260807 22:48:13.752875 30711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44f29a09a4f74e7bb3a420b819342612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } } }
I20260807 22:48:13.752990 30711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612 [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:13.753505 30674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44f29a09a4f74e7bb3a420b819342612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc58513e85c45d08658534345b25c91" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46597 } } peers { permanent_uuid: "9e71f9efe3654d57a9c746631f09287a" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40979 } } peers { permanent_uuid: "44f29a09a4f74e7bb3a420b819342612" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 39781 } } }
I20260807 22:48:13.753597 30674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:48:13.753997 30730 mvcc.cc:205] Tried to move back new op lower bound from 7316041292762877952 to 7316041292716384256. Current Snapshot: MvccSnapshot[applied={T|T < 7316041292762877952}]
I20260807 22:48:13.757272 30726 catalog_manager.cc:1347] Generated new cluster ID: 6e06db5d5cbe484c931815cc1eedcbf2
I20260807 22:48:13.757337 30726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:48:13.770480 30726 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:48:13.773146 30726 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:48:13.790320 30726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44f29a09a4f74e7bb3a420b819342612: Generated new TSK 0
I20260807 22:48:13.790683 30726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:48:14.682726 30686 catalog_manager.cc:1259] Loaded cluster ID: 6e06db5d5cbe484c931815cc1eedcbf2
I20260807 22:48:14.682850 30686 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a: loading cluster ID for follower catalog manager: success
I20260807 22:48:14.683705 30686 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a: acquiring CA information for follower catalog manager: success
I20260807 22:48:14.684098 30686 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9e71f9efe3654d57a9c746631f09287a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 22:48:14.697513 30706 catalog_manager.cc:1259] Loaded cluster ID: 6e06db5d5cbe484c931815cc1eedcbf2
I20260807 22:48:14.697606 30706 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91: loading cluster ID for follower catalog manager: success
I20260807 22:48:14.698526 30706 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91: acquiring CA information for follower catalog manager: success
I20260807 22:48:14.698879 30706 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6cc58513e85c45d08658534345b25c91: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 22:48:14.852850 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 30440
I20260807 22:48:14.860345 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 30517
I20260807 22:48:14.867087 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 30596
2026-08-07T22:48:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3178 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:48:14 dist-test-slave-rgb1 krb5kdc[30745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:48:14 dist-test-slave-rgb1 krb5kdc[30745](info): set up 2 sockets
Aug 07 22:48:14 dist-test-slave-rgb1 krb5kdc[30745](info): commencing operation
krb5kdc: starting...
W20260807 22:48:16.037352 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:48:16 dist-test-slave-rgb1 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:48:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:48:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:48:16.184109 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:33935
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:33935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:16.278508 30758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:16.278762 30758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:16.278800 30758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:16.279968 30758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:48:16.280036 30758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:16.280061 30758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:16.280088 30758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:48:16.280121 30758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:48:16.281713 30758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:33935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:33935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:16.282014 30758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:16.282272 30758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:16.287436 30766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:16.287429 30763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:16.287461 30764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:16.288003 30758 server_base.cc:1034] running on GCE node
I20260807 22:48:16.288316 30758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:16.288708 30758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:16.289894 30758 hybrid_clock.cc:648] HybridClock initialized: now 1786142896289875 us; error 53 us; skew 500 ppm
Aug 07 22:48:16 dist-test-slave-rgb1 krb5kdc[30745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142896, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:16.292835 30758 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:48:16.293431 30758 webserver.cc:460] Webserver started at http://127.16.98.190:37605/ using document root <none> and password file <none>
I20260807 22:48:16.293655 30758 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:16.293699 30758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:16.293795 30758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:48:16.294842 30758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "fa750bb9080f46bda98070ddecfd9e8f"
format_stamp: "Formatted at 2026-08-07 22:48:16 on dist-test-slave-rgb1"
server_key: "e9fcd10ed7824b22fb991aa2f241163d"
server_key_iv: "a691b648abb163df9c3fd66ac9b3a3cc"
server_key_version: "encryptionkey@0"
I20260807 22:48:16.295156 30758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "fa750bb9080f46bda98070ddecfd9e8f"
format_stamp: "Formatted at 2026-08-07 22:48:16 on dist-test-slave-rgb1"
server_key: "e9fcd10ed7824b22fb991aa2f241163d"
server_key_iv: "a691b648abb163df9c3fd66ac9b3a3cc"
server_key_version: "encryptionkey@0"
I20260807 22:48:16.296957 30758 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260807 22:48:16.297873 30773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:16.298149 30758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:16.298283 30758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "fa750bb9080f46bda98070ddecfd9e8f"
format_stamp: "Formatted at 2026-08-07 22:48:16 on dist-test-slave-rgb1"
server_key: "e9fcd10ed7824b22fb991aa2f241163d"
server_key_iv: "a691b648abb163df9c3fd66ac9b3a3cc"
server_key_version: "encryptionkey@0"
I20260807 22:48:16.298362 30758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:16.323550 30758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:16.326282 30758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:16.326418 30758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:16.330940 30758 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:33935
I20260807 22:48:16.331058 30827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:33935 every 8 connection(s)
I20260807 22:48:16.331440 30758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:48:16.332393 30828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:16.334807 30828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f: Bootstrap starting.
I20260807 22:48:16.335508 30828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:16.335808 30828 log.cc:824] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:16.336599 30828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f: No bootstrap required, opened a new log
I20260807 22:48:16.338315 30828 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa750bb9080f46bda98070ddecfd9e8f" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 33935 } }
I20260807 22:48:16.338486 30828 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:16.338529 30828 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa750bb9080f46bda98070ddecfd9e8f, State: Initialized, Role: FOLLOWER
I20260807 22:48:16.338712 30828 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa750bb9080f46bda98070ddecfd9e8f" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 33935 } }
I20260807 22:48:16.338814 30828 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:48:16.338860 30828 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:48:16.338912 30828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:16.339530 30828 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa750bb9080f46bda98070ddecfd9e8f" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 33935 } }
I20260807 22:48:16.339658 30828 leader_election.cc:302] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa750bb9080f46bda98070ddecfd9e8f; no voters:
I20260807 22:48:16.339856 30828 leader_election.cc:288] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:48:16.339905 30831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:48:16.340149 30831 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [term 1 LEADER]: Becoming Leader. State: Replica: fa750bb9080f46bda98070ddecfd9e8f, State: Running, Role: LEADER
I20260807 22:48:16.340147 30828 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:48:16.340260 30831 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa750bb9080f46bda98070ddecfd9e8f" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 33935 } }
I20260807 22:48:16.340723 30832 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa750bb9080f46bda98070ddecfd9e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa750bb9080f46bda98070ddecfd9e8f" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 33935 } } }
I20260807 22:48:16.340819 30832 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:16.340860 30833 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa750bb9080f46bda98070ddecfd9e8f. Latest consensus state: current_term: 1 leader_uuid: "fa750bb9080f46bda98070ddecfd9e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa750bb9080f46bda98070ddecfd9e8f" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 33935 } } }
I20260807 22:48:16.340931 30833 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:16.341215 30840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:48:16.341543 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 30758
I20260807 22:48:16.341670 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:48:16.341760 30840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:48:16.341924 16778 external_mini_cluster.cc:1428] Setting key c3d6fb24fda86108d1b33088d86b3c17
I20260807 22:48:16.343461 30840 catalog_manager.cc:1347] Generated new cluster ID: dda055985a2343369aa2b896a157a14e
I20260807 22:48:16.343525 30840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 22:48:16 dist-test-slave-rgb1 krb5kdc[30745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:16.353050 30840 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:48:16.353756 30840 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:48:16.354151 30850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:16.342397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.16.98.190:33935)
0807 22:48:16.342587 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:16.342597 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:16.342630 (+ 33us) server_negotiation.cc:407] Connection header received
0807 22:48:16.342747 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:16.342752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:16.342794 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:16.342950 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:16.343751 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:16.345189 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:16.346284 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:16.346940 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:16.347048 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:16.350081 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:16.350086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:16.350087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:16.350101 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:16.352131 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:16.352601 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:16.352604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:16.352605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:16.352758 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:16.353040 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:16.353041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:16.353041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:16.353336 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:16.353494 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:16.353764 (+ 270us) server_negotiation.cc:299] Negotiation successful
0807 22:48:16.353926 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20260807 22:48:16.355445 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:48:16.369184 30840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa750bb9080f46bda98070ddecfd9e8f: Generated new TSK 0
I20260807 22:48:16.369437 30840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:48:16.409865 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 30758
2026-08-07T22:48:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1542 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:48:16 dist-test-slave-rgb1 krb5kdc[30855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:48:16 dist-test-slave-rgb1 krb5kdc[30855](info): set up 2 sockets
Aug 07 22:48:16 dist-test-slave-rgb1 krb5kdc[30855](info): commencing operation
krb5kdc: starting...
W20260807 22:48:17.550839 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:48:17 dist-test-slave-rgb1 krb5kdc[30855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:48:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:48:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:48:17.700801 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43929
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:43929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:17.791054 30868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:17.791335 30868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:17.791378 30868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:17.792732 30868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:48:17.792807 30868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:17.792832 30868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:17.792855 30868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:48:17.792883 30868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:48:17.794818 30868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:43929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:17.795164 30868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:17.795428 30868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:17.801507 30874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:17.801524 30868 server_base.cc:1034] running on GCE node
W20260807 22:48:17.801555 30873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:17.801716 30876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:17.802290 30868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:17.802917 30868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:17.804165 30868 hybrid_clock.cc:648] HybridClock initialized: now 1786142897804133 us; error 53 us; skew 500 ppm
Aug 07 22:48:17 dist-test-slave-rgb1 krb5kdc[30855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142897, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:17.807356 30868 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:48:17.807902 30868 webserver.cc:460] Webserver started at http://127.16.98.190:32835/ using document root <none> and password file <none>
I20260807 22:48:17.808166 30868 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:17.808236 30868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:17.808393 30868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:48:17.809603 30868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "c51bff78f1cf4e4d82ff9fb9c632de52"
format_stamp: "Formatted at 2026-08-07 22:48:17 on dist-test-slave-rgb1"
server_key: "6ca1ae94f4683fe7a302a5b5f01f22df"
server_key_iv: "c3a2051eeb2adfa09c1b56e1b6741b26"
server_key_version: "encryptionkey@0"
I20260807 22:48:17.809991 30868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "c51bff78f1cf4e4d82ff9fb9c632de52"
format_stamp: "Formatted at 2026-08-07 22:48:17 on dist-test-slave-rgb1"
server_key: "6ca1ae94f4683fe7a302a5b5f01f22df"
server_key_iv: "c3a2051eeb2adfa09c1b56e1b6741b26"
server_key_version: "encryptionkey@0"
I20260807 22:48:17.811445 30868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:48:17.812345 30883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:17.812597 30868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:17.812711 30868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "c51bff78f1cf4e4d82ff9fb9c632de52"
format_stamp: "Formatted at 2026-08-07 22:48:17 on dist-test-slave-rgb1"
server_key: "6ca1ae94f4683fe7a302a5b5f01f22df"
server_key_iv: "c3a2051eeb2adfa09c1b56e1b6741b26"
server_key_version: "encryptionkey@0"
I20260807 22:48:17.812808 30868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:17.827720 30868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:17.830606 30868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:17.830897 30868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:17.836045 30868 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:43929
I20260807 22:48:17.836102 30935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:43929 every 8 connection(s)
I20260807 22:48:17.836517 30868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:48:17.837565 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 30868
I20260807 22:48:17.837743 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:48:17.837962 30936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:17.838027 16778 external_mini_cluster.cc:1428] Setting key 468b84bede4215cd89288f9fda3508f5
I20260807 22:48:17.841768 30936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: Bootstrap starting.
I20260807 22:48:17.842489 30936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:17.842841 30936 log.cc:824] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:17.843595 30936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: No bootstrap required, opened a new log
I20260807 22:48:17.845580 30936 raft_consensus.cc:348] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } }
I20260807 22:48:17.845759 30936 raft_consensus.cc:374] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:17.845810 30936 raft_consensus.cc:729] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c51bff78f1cf4e4d82ff9fb9c632de52, State: Initialized, Role: FOLLOWER
Aug 07 22:48:17 dist-test-slave-rgb1 krb5kdc[30855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:17.845986 30936 consensus_queue.cc:260] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } }
I20260807 22:48:17.846123 30936 raft_consensus.cc:388] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:48:17.846169 30936 raft_consensus.cc:482] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:48:17.846221 30936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:17.846853 30936 raft_consensus.cc:504] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } }
I20260807 22:48:17.846988 30936 leader_election.cc:302] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c51bff78f1cf4e4d82ff9fb9c632de52; no voters:
I20260807 22:48:17.847198 30936 leader_election.cc:288] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:48:17.847291 30941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:48:17.847476 30941 raft_consensus.cc:686] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 1 LEADER]: Becoming Leader. State: Replica: c51bff78f1cf4e4d82ff9fb9c632de52, State: Running, Role: LEADER
I20260807 22:48:17.847519 30936 sys_catalog.cc:564] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:48:17.847587 30941 consensus_queue.cc:237] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } }
I20260807 22:48:17.848026 30942 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } } }
I20260807 22:48:17.848017 30941 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c51bff78f1cf4e4d82ff9fb9c632de52. Latest consensus state: current_term: 1 leader_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } } }
I20260807 22:48:17.848196 30942 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:17.848213 30941 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [sys.catalog]: This master's current role is: LEADER
W20260807 22:48:17.849826 30957 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:48:17.849898 30957 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:48:17.849961 30953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:48:17.850139 30953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:48:17.851581 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:17.838721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.16.98.190:43929)
0807 22:48:17.839014 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:17.839025 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:17.839062 (+ 37us) server_negotiation.cc:407] Connection header received
0807 22:48:17.839593 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:17.839600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:17.839653 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:17.839836 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:17.840693 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:17.841759 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:17.842748 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:17.843362 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:17.843526 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:17.846630 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:17.846634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:17.846635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:17.846647 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:17.849611 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:17.850111 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:17.850114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:17.850115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:17.850263 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:17.850500 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:17.850501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:17.850501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:17.850785 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:17.850957 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:17.851157 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 22:48:17.851323 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260807 22:48:17.852352 30953 catalog_manager.cc:1347] Generated new cluster ID: be3e6ad8537e4970a8af55a34edb875f
I20260807 22:48:17.852468 30953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:48:17.871407 30953 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:48:17.872107 30953 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:48:17.890129 30953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: Generated new TSK 0
I20260807 22:48:17.890409 30953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:48:17.938745 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43929
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:18.079681 30963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:18.080137 30963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:18.080194 30963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:18.082295 30963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:18.082367 30963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:18.082402 30963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:48:18.084509 30963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:18.084824 30963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:18.085057 30963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:18.091915 30969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:18.091915 30971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:18.091949 30968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:18.092530 30963 server_base.cc:1034] running on GCE node
I20260807 22:48:18.092828 30963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:18.093510 30963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:18.095286 30963 hybrid_clock.cc:648] HybridClock initialized: now 1786142898095230 us; error 286 us; skew 500 ppm
Aug 07 22:48:18 dist-test-slave-rgb1 krb5kdc[30855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:18.100132 30963 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:48:18.100872 30963 webserver.cc:460] Webserver started at http://127.16.98.129:46469/ using document root <none> and password file <none>
I20260807 22:48:18.101434 30963 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:18.101548 30963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:18.101738 30963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:48:18.103241 30963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "77016cd86efa44cfb5534027d2120355"
format_stamp: "Formatted at 2026-08-07 22:48:18 on dist-test-slave-rgb1"
server_key: "2ba65eb4ec53c414de6f81938b64528f"
server_key_iv: "aa0344ba5453ccd854778deb178b05ce"
server_key_version: "encryptionkey@0"
I20260807 22:48:18.103919 30963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "77016cd86efa44cfb5534027d2120355"
format_stamp: "Formatted at 2026-08-07 22:48:18 on dist-test-slave-rgb1"
server_key: "2ba65eb4ec53c414de6f81938b64528f"
server_key_iv: "aa0344ba5453ccd854778deb178b05ce"
server_key_version: "encryptionkey@0"
I20260807 22:48:18.105783 30963 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260807 22:48:18.107545 30978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:18.108201 30963 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:48:18.108441 30963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "77016cd86efa44cfb5534027d2120355"
format_stamp: "Formatted at 2026-08-07 22:48:18 on dist-test-slave-rgb1"
server_key: "2ba65eb4ec53c414de6f81938b64528f"
server_key_iv: "aa0344ba5453ccd854778deb178b05ce"
server_key_version: "encryptionkey@0"
I20260807 22:48:18.108857 30963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:18.128310 30963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:18.132170 30963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:18.132393 30963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:18.132884 30963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:18.133325 30963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:18.133388 30963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:18.133426 30963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:18.133453 30963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:18.141177 30963 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:33263
I20260807 22:48:18.141320 31092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:33263 every 8 connection(s)
I20260807 22:48:18.141635 30963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:48:18.149737 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 30963
I20260807 22:48:18.149923 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:48:18.150200 16778 external_mini_cluster.cc:1428] Setting key 018c749ec679ee3ef445abb9a14e78a5
Aug 07 22:48:18 dist-test-slave-rgb1 krb5kdc[30855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:18.156461 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.142547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:58519 (local address 127.16.98.190:43929)
0807 22:48:18.142734 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:18.142736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:18.142790 (+ 54us) server_negotiation.cc:407] Connection header received
0807 22:48:18.143348 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:18.143350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:18.143362 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:18.143446 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:18.144807 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.145621 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.147049 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.147398 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.147645 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.151113 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:18.151115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:18.151115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:18.151123 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:18.153286 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:18.154369 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:18.154370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:18.154371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:18.154457 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:18.155196 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:18.155197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:18.155198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:18.155474 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:18.155716 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:18.156263 (+ 547us) server_negotiation.cc:299] Negotiation successful
0807 22:48:18.156384 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260807 22:48:18.156783 31095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.142500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:58519)
0807 22:48:18.142666 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:18.142712 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:18.143221 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:18.143570 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:18.143575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:18.144297 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:18.144650 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.144658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.145769 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.145771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.146843 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.146847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.147680 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.147681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.147814 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.148610 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:18.148621 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:18.150980 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:18.153470 (+ 2490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:18.153499 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:18.153503 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:18.154170 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:18.154570 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:18.154571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:18.154572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:18.154988 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:18.155699 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:18.155711 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:18.156050 (+ 339us) client_negotiation.cc:769] Sending connection context
0807 22:48:18.156205 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 22:48:18.156436 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260807 22:48:18.157508 31093 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43929
I20260807 22:48:18.157850 31093 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:18.158272 31093 heartbeater.cc:511] Master 127.16.98.190:43929 requested a full tablet report, sending...
I20260807 22:48:18.159001 30900 ts_manager.cc:194] Registered new tserver with Master: 77016cd86efa44cfb5534027d2120355 (127.16.98.129:33263)
I20260807 22:48:18.159740 30900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:58519
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:48:18.194310 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43929
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:18.294323 31098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:18.294867 31098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:18.294960 31098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:18.296427 31098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:18.296526 31098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:18.296612 31098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:48:18.298597 31098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:18.298940 31098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:18.299225 31098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:18.305461 31106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:18.305528 31104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:18.305773 31098 server_base.cc:1034] running on GCE node
W20260807 22:48:18.305466 31103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:18.306272 31098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:18.306602 31098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:18.307813 31098 hybrid_clock.cc:648] HybridClock initialized: now 1786142898307771 us; error 65 us; skew 500 ppm
Aug 07 22:48:18 dist-test-slave-rgb1 krb5kdc[30855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:18.311290 31098 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:48:18.311925 31098 webserver.cc:460] Webserver started at http://127.16.98.130:42805/ using document root <none> and password file <none>
I20260807 22:48:18.312255 31098 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:18.312309 31098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:18.312474 31098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:48:18.313771 31098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "f42693cabf23402eb345705c414c36db"
format_stamp: "Formatted at 2026-08-07 22:48:18 on dist-test-slave-rgb1"
server_key: "58aacebf8ca45ca010530023af3cb8f0"
server_key_iv: "4658b65b8b7c4ada247583709f1f89b1"
server_key_version: "encryptionkey@0"
I20260807 22:48:18.314277 31098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "f42693cabf23402eb345705c414c36db"
format_stamp: "Formatted at 2026-08-07 22:48:18 on dist-test-slave-rgb1"
server_key: "58aacebf8ca45ca010530023af3cb8f0"
server_key_iv: "4658b65b8b7c4ada247583709f1f89b1"
server_key_version: "encryptionkey@0"
I20260807 22:48:18.316299 31098 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 22:48:18.317785 31113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:18.318252 31098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:48:18.318364 31098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "f42693cabf23402eb345705c414c36db"
format_stamp: "Formatted at 2026-08-07 22:48:18 on dist-test-slave-rgb1"
server_key: "58aacebf8ca45ca010530023af3cb8f0"
server_key_iv: "4658b65b8b7c4ada247583709f1f89b1"
server_key_version: "encryptionkey@0"
I20260807 22:48:18.318425 31098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:18.337411 31098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:18.341276 31098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:18.341598 31098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:18.341881 31098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:18.342343 31098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:18.342397 31098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:18.342581 31098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:18.342629 31098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:18.350636 31098 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:37707
I20260807 22:48:18.350732 31228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:37707 every 8 connection(s)
I20260807 22:48:18.351080 31098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:48:18.352910 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 31098
I20260807 22:48:18.353037 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:48:18.353227 16778 external_mini_cluster.cc:1428] Setting key 7280e495a68e768a3a792a09851692da
Aug 07 22:48:18 dist-test-slave-rgb1 krb5kdc[30855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:18.363852 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.351956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:43813 (local address 127.16.98.190:43929)
0807 22:48:18.352016 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:18.352019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:18.352332 (+ 313us) server_negotiation.cc:407] Connection header received
0807 22:48:18.353131 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:18.353134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:18.353230 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:18.353389 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:18.355056 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.355631 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.357108 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.357322 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.357393 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.360606 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:18.360608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:18.360608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:18.360618 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:18.362142 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:18.362773 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:18.362775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:18.362775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:18.362842 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:18.363121 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:18.363122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:18.363123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:18.363256 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:18.363339 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:18.363648 (+ 309us) server_negotiation.cc:299] Negotiation successful
0807 22:48:18.363745 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 22:48:18.364179 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.351956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:43813)
0807 22:48:18.352151 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:18.352206 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:18.352872 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:18.353563 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:18.353571 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:18.354654 (+ 1083us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:18.354897 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.354906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.355768 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.355769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.356937 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.356941 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.357426 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.357428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.357601 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.358541 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:18.358553 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:18.360459 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:18.362264 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:18.362275 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:18.362277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:18.362651 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:18.362924 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:18.362925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:18.362925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:18.363048 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:18.363329 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:18.363336 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:18.363499 (+ 163us) client_negotiation.cc:769] Sending connection context
0807 22:48:18.363628 (+ 129us) client_negotiation.cc:240] Negotiation successful
0807 22:48:18.363822 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260807 22:48:18.364966 31229 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43929
I20260807 22:48:18.365272 31229 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
I20260807 22:48:18.365628 31229 heartbeater.cc:511] Master 127.16.98.190:43929 requested a full tablet report, sending...
I20260807 22:48:18.366400 30900 ts_manager.cc:194] Registered new tserver with Master: f42693cabf23402eb345705c414c36db (127.16.98.130:37707)
I20260807 22:48:18.366963 30900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:43813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:48:18.393612 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43929
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:18.504776 31234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:18.505057 31234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:18.505091 31234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:18.507211 31234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:18.507306 31234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:18.507350 31234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:48:18.509857 31234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:18.510138 31234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:18.510351 31234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:18.517804 31242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:18.517966 31240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:18.517804 31239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:18.518518 31234 server_base.cc:1034] running on GCE node
I20260807 22:48:18.518796 31234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:18.519800 31234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:18.521291 31234 hybrid_clock.cc:648] HybridClock initialized: now 1786142898521265 us; error 63 us; skew 500 ppm
Aug 07 22:48:18 dist-test-slave-rgb1 krb5kdc[30855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:18.525374 31234 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:48:18.526057 31234 webserver.cc:460] Webserver started at http://127.16.98.131:44363/ using document root <none> and password file <none>
I20260807 22:48:18.526345 31234 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:18.526425 31234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:18.526620 31234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:48:18.527910 31234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "4c93f70b4fe645118f3d1207b5b13bbe"
format_stamp: "Formatted at 2026-08-07 22:48:18 on dist-test-slave-rgb1"
server_key: "451149b16629902ff49a3802c4603e0e"
server_key_iv: "f1b9086ffa1b0c1444f590ba8ba27987"
server_key_version: "encryptionkey@0"
I20260807 22:48:18.528389 31234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "4c93f70b4fe645118f3d1207b5b13bbe"
format_stamp: "Formatted at 2026-08-07 22:48:18 on dist-test-slave-rgb1"
server_key: "451149b16629902ff49a3802c4603e0e"
server_key_iv: "f1b9086ffa1b0c1444f590ba8ba27987"
server_key_version: "encryptionkey@0"
I20260807 22:48:18.530643 31234 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260807 22:48:18.532760 31249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:18.534984 31234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 22:48:18.535223 31234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "4c93f70b4fe645118f3d1207b5b13bbe"
format_stamp: "Formatted at 2026-08-07 22:48:18 on dist-test-slave-rgb1"
server_key: "451149b16629902ff49a3802c4603e0e"
server_key_iv: "f1b9086ffa1b0c1444f590ba8ba27987"
server_key_version: "encryptionkey@0"
I20260807 22:48:18.535341 31234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:18.553139 31234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:18.556311 31234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:18.556522 31234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:18.556882 31234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:18.557293 31234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:18.557351 31234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:18.557397 31234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:18.557425 31234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:18.569212 31234 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:39995
I20260807 22:48:18.569260 31364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:39995 every 8 connection(s)
I20260807 22:48:18.569643 31234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:48:18.572898 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 31234
I20260807 22:48:18.573212 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:48:18.573437 16778 external_mini_cluster.cc:1428] Setting key 6f3b639b4c03ba05deb01228ee4a1424
Aug 07 22:48:18 dist-test-slave-rgb1 krb5kdc[30855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:18.586835 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.570683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:59917 (local address 127.16.98.190:43929)
0807 22:48:18.570747 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:18.570750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:18.571275 (+ 525us) server_negotiation.cc:407] Connection header received
0807 22:48:18.571854 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:18.571856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:18.571870 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:18.571919 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:18.573837 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.575130 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.577801 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.578037 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.578150 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.582396 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:18.582399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:18.582399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:18.582409 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:18.584469 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:18.585232 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:18.585239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:18.585239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:18.585390 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:18.586018 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:18.586020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:18.586020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:18.586198 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:18.586287 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:18.586627 (+ 340us) server_negotiation.cc:299] Negotiation successful
0807 22:48:18.586728 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260807 22:48:18.587136 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.570637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:59917)
0807 22:48:18.571133 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:18.571174 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:18.571733 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:18.572008 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:18.572012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:18.573419 (+ 1407us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:18.573691 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.573699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.575246 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.575248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.577658 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.577663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.578159 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.578160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.578325 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.579395 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:18.579409 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:18.582230 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:18.584640 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:18.584654 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:18.584661 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:18.584980 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:18.585769 (+ 789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:18.585770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:18.585770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:18.585924 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:18.586287 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:18.586296 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:18.586461 (+ 165us) client_negotiation.cc:769] Sending connection context
0807 22:48:18.586584 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 22:48:18.586788 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":219,"threads_started":1}
I20260807 22:48:18.587900 31365 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43929
I20260807 22:48:18.588297 31365 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:18.588629 31365 heartbeater.cc:511] Master 127.16.98.190:43929 requested a full tablet report, sending...
I20260807 22:48:18.589509 30899 ts_manager.cc:194] Registered new tserver with Master: 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995)
I20260807 22:48:18.590261 30899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:59917
I20260807 22:48:18.599911 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:48:18.609474 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.601669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53614 (local address 127.16.98.190:43929)
0807 22:48:18.601758 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:18.601760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:18.601898 (+ 138us) server_negotiation.cc:407] Connection header received
0807 22:48:18.601992 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:18.601993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:18.602004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:18.602137 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:18.602823 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.603370 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.604205 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.604430 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.604515 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.606046 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:18.606048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:18.606048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:18.606058 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:18.607626 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:18.608103 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:18.608105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:18.608105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:18.608173 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:18.608520 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:18.608522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:18.608522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:18.608888 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:18.609045 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:18.609203 (+ 158us) server_negotiation.cc:299] Negotiation successful
0807 22:48:18.609317 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 22:48:18.612186 30899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:48:18.614202 30899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:48:18.622757 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.617280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:33263 (local address 127.0.0.1:55364)
0807 22:48:18.617614 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:18.617645 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:18.617715 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:18.618031 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:18.618032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:18.618035 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:18.618122 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.618125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.619393 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.619394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.621219 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.621223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.622388 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.622389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.622503 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.622523 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 22:48:18.622600 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 22:48:18.622657 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":3,"thread_start_us":98,"threads_started":1}
I20260807 22:48:18.623111 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.617416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.16.98.129:33263)
0807 22:48:18.617829 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:18.617832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:18.617841 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:48:18.617891 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:18.617892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:18.617924 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:18.618025 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:18.618211 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.619227 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.621328 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.622285 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.622362 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.622431 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:18.622646 (+ 215us) server_negotiation.cc:299] Negotiation successful
0807 22:48:18.622730 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":100,"threads_started":1}
I20260807 22:48:18.623445 31377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.617280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:39995 (local address 127.0.0.1:48216)
0807 22:48:18.617598 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:18.617645 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:18.617715 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:18.617954 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:18.617957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:18.617960 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:18.618045 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.618048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.620423 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.620424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.622024 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.622027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.623198 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.623199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.623263 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.623267 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:48:18.623327 (+ 60us) client_negotiation.cc:240] Negotiation successful
0807 22:48:18.623374 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":3,"thread_start_us":53,"threads_started":1}
I20260807 22:48:18.623875 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.617383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.16.98.131:39995)
0807 22:48:18.617688 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:18.617693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:18.617707 (+ 14us) server_negotiation.cc:407] Connection header received
0807 22:48:18.617845 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:18.617847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:18.617860 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:18.617920 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:18.618146 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.619462 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.622136 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.623107 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.623197 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.623254 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:18.623362 (+ 108us) server_negotiation.cc:299] Negotiation successful
0807 22:48:18.623405 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":99,"threads_started":1}
I20260807 22:48:18.624125 31380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.617498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:37707 (local address 127.0.0.1:58464)
0807 22:48:18.618026 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:18.618036 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:18.618086 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:18.618360 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:18.618361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:18.618363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:18.618407 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.618408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.621307 (+ 2899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.621308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.622562 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.622565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.623839 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.623841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.623910 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.623915 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:48:18.623983 (+ 68us) client_negotiation.cc:240] Negotiation successful
0807 22:48:18.624040 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260807 22:48:18.624161 31026 tablet_service.cc:1467] Processing CreateTablet for tablet 630d66bbf28f4a958deb4e2e9bf80b8f (DEFAULT_TABLE table=test-table [id=59a444a2af8f44e4a21c6e07a28b718e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:48:18.624488 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.617642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58464 (local address 127.16.98.130:37707)
0807 22:48:18.617800 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:18.617803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:18.618146 (+ 343us) server_negotiation.cc:407] Connection header received
0807 22:48:18.618195 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:18.618198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:18.618214 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:18.618284 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:18.618484 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.621150 (+ 2666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.622698 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.623599 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.623735 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.623797 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:18.624082 (+ 285us) server_negotiation.cc:299] Negotiation successful
0807 22:48:18.624145 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260807 22:48:18.624646 31026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630d66bbf28f4a958deb4e2e9bf80b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:18.625363 31298 tablet_service.cc:1467] Processing CreateTablet for tablet 630d66bbf28f4a958deb4e2e9bf80b8f (DEFAULT_TABLE table=test-table [id=59a444a2af8f44e4a21c6e07a28b718e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:48:18.625361 31161 tablet_service.cc:1467] Processing CreateTablet for tablet 630d66bbf28f4a958deb4e2e9bf80b8f (DEFAULT_TABLE table=test-table [id=59a444a2af8f44e4a21c6e07a28b718e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:48:18.625717 31298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630d66bbf28f4a958deb4e2e9bf80b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:18.625715 31161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630d66bbf28f4a958deb4e2e9bf80b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:18.628947 31382 tablet_bootstrap.cc:492] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: Bootstrap starting.
I20260807 22:48:18.629593 31383 tablet_bootstrap.cc:492] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db: Bootstrap starting.
I20260807 22:48:18.630079 31382 tablet_bootstrap.cc:654] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:18.630201 31383 tablet_bootstrap.cc:654] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:18.630805 31383 log.cc:824] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:18.630928 31382 log.cc:824] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:18.631224 31384 tablet_bootstrap.cc:492] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe: Bootstrap starting.
I20260807 22:48:18.631786 31382 tablet_bootstrap.cc:492] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: No bootstrap required, opened a new log
I20260807 22:48:18.631863 31384 tablet_bootstrap.cc:654] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:18.631883 31382 ts_tablet_manager.cc:1397] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:48:18.632386 31384 log.cc:824] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:18.632992 31383 tablet_bootstrap.cc:492] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db: No bootstrap required, opened a new log
I20260807 22:48:18.633131 31383 ts_tablet_manager.cc:1397] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 22:48:18.633435 31384 tablet_bootstrap.cc:492] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe: No bootstrap required, opened a new log
I20260807 22:48:18.633524 31384 ts_tablet_manager.cc:1397] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260807 22:48:18.634392 31382 raft_consensus.cc:348] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } }
I20260807 22:48:18.634740 31382 raft_consensus.cc:374] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:18.634793 31382 raft_consensus.cc:729] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77016cd86efa44cfb5534027d2120355, State: Initialized, Role: FOLLOWER
I20260807 22:48:18.634975 31382 consensus_queue.cc:260] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } }
I20260807 22:48:18.635326 31382 ts_tablet_manager.cc:1428] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260807 22:48:18.635517 31093 heartbeater.cc:503] Master 127.16.98.190:43929 was elected leader, sending a full tablet report...
I20260807 22:48:18.636171 31384 raft_consensus.cc:348] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } }
I20260807 22:48:18.636195 31383 raft_consensus.cc:348] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } }
I20260807 22:48:18.636372 31383 raft_consensus.cc:374] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:18.636373 31384 raft_consensus.cc:374] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:18.636421 31383 raft_consensus.cc:729] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42693cabf23402eb345705c414c36db, State: Initialized, Role: FOLLOWER
I20260807 22:48:18.636421 31384 raft_consensus.cc:729] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c93f70b4fe645118f3d1207b5b13bbe, State: Initialized, Role: FOLLOWER
I20260807 22:48:18.636590 31383 consensus_queue.cc:260] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } }
I20260807 22:48:18.636607 31384 consensus_queue.cc:260] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } }
I20260807 22:48:18.636938 31384 ts_tablet_manager.cc:1428] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260807 22:48:18.636950 31383 ts_tablet_manager.cc:1428] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260807 22:48:18.636988 31229 heartbeater.cc:503] Master 127.16.98.190:43929 was elected leader, sending a full tablet report...
I20260807 22:48:18.637107 31365 heartbeater.cc:503] Master 127.16.98.190:43929 was elected leader, sending a full tablet report...
W20260807 22:48:18.642444 31094 tablet.cc:2369] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:48:18.659438 31389 raft_consensus.cc:482] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:48:18.659614 31389 raft_consensus.cc:504] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } }
I20260807 22:48:18.660142 31389 leader_election.cc:288] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77016cd86efa44cfb5534027d2120355 (127.16.98.129:33263), f42693cabf23402eb345705c414c36db (127.16.98.130:37707)
I20260807 22:48:18.663748 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.660440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:33263 (local address 127.16.98.131:40891)
0807 22:48:18.660588 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:18.660602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:18.660671 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:18.660852 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:18.660853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:18.660855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:18.660947 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.660950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.661685 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.661686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.662737 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.662740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.663403 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.663404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.663504 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.663507 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:48:18.663584 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 22:48:18.663635 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":1}
I20260807 22:48:18.663756 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.660607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:40891 (local address 127.16.98.129:33263)
0807 22:48:18.660659 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:18.660661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:18.660673 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:48:18.660743 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:18.660744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:18.660754 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:18.660829 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:18.661108 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.661595 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.662873 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.663300 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.663394 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.663406 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:18.663606 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 22:48:18.663649 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 22:48:18.664014 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.660688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:37707 (local address 127.16.98.131:35841)
0807 22:48:18.660919 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:18.660928 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:18.661209 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:18.661404 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:18.661405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:18.661407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:18.661503 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.661506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.662383 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.662384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.663133 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:18.663135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.663702 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.663703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:18.663780 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.663783 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:48:18.663859 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 22:48:18.663911 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":156,"threads_started":1}
I20260807 22:48:18.664142 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.660765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:35841 (local address 127.16.98.130:37707)
0807 22:48:18.660844 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:18.660847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:18.661185 (+ 338us) server_negotiation.cc:407] Connection header received
0807 22:48:18.661308 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:18.661309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:18.661325 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:18.661397 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:18.661578 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.662275 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.663224 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.663615 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.663701 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.663710 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:18.663993 (+ 283us) server_negotiation.cc:299] Negotiation successful
0807 22:48:18.664056 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 22:48:18.664166 31046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630d66bbf28f4a958deb4e2e9bf80b8f" candidate_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77016cd86efa44cfb5534027d2120355" is_pre_election: true
I20260807 22:48:18.664355 31046 raft_consensus.cc:2454] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c93f70b4fe645118f3d1207b5b13bbe in term 0.
I20260807 22:48:18.664450 31181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630d66bbf28f4a958deb4e2e9bf80b8f" candidate_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42693cabf23402eb345705c414c36db" is_pre_election: true
I20260807 22:48:18.664636 31181 raft_consensus.cc:2454] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c93f70b4fe645118f3d1207b5b13bbe in term 0.
I20260807 22:48:18.664674 31251 leader_election.cc:302] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c93f70b4fe645118f3d1207b5b13bbe, 77016cd86efa44cfb5534027d2120355; no voters:
I20260807 22:48:18.664831 31389 raft_consensus.cc:2781] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:48:18.664919 31389 raft_consensus.cc:482] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:48:18.665050 31389 raft_consensus.cc:3037] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:18.665905 31389 raft_consensus.cc:504] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } }
I20260807 22:48:18.666064 31389 leader_election.cc:288] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [CANDIDATE]: Term 1 election: Requested vote from peers 77016cd86efa44cfb5534027d2120355 (127.16.98.129:33263), f42693cabf23402eb345705c414c36db (127.16.98.130:37707)
I20260807 22:48:18.666313 31046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630d66bbf28f4a958deb4e2e9bf80b8f" candidate_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77016cd86efa44cfb5534027d2120355"
I20260807 22:48:18.666345 31181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630d66bbf28f4a958deb4e2e9bf80b8f" candidate_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42693cabf23402eb345705c414c36db"
I20260807 22:48:18.666458 31181 raft_consensus.cc:3037] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:18.666446 31046 raft_consensus.cc:3037] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:18.667402 31046 raft_consensus.cc:2454] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c93f70b4fe645118f3d1207b5b13bbe in term 1.
I20260807 22:48:18.667613 31181 raft_consensus.cc:2454] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c93f70b4fe645118f3d1207b5b13bbe in term 1.
I20260807 22:48:18.667655 31251 leader_election.cc:302] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c93f70b4fe645118f3d1207b5b13bbe, 77016cd86efa44cfb5534027d2120355; no voters:
I20260807 22:48:18.667773 31389 raft_consensus.cc:2781] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:48:18.667965 31389 raft_consensus.cc:686] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [term 1 LEADER]: Becoming Leader. State: Replica: 4c93f70b4fe645118f3d1207b5b13bbe, State: Running, Role: LEADER
I20260807 22:48:18.668183 31389 consensus_queue.cc:237] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } }
I20260807 22:48:18.669413 30900 catalog_manager.cc:5495] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131). New cstate: current_term: 1 leader_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } health_report { overall_health: UNKNOWN } } }
I20260807 22:48:18.686931 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.683157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.16.98.131:39995)
0807 22:48:18.683230 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:18.683232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:18.683241 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:48:18.683367 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:18.683368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:18.683388 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:18.683487 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:48:18.683869 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.684611 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.685692 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:18.685927 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:18.686010 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:18.686312 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:48:18.686439 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:48:18.686659 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:18.686803 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 22:48:18.686832 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260807 22:48:18.689872 31181 raft_consensus.cc:1261] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db [term 1 FOLLOWER]: Refusing update from remote peer 4c93f70b4fe645118f3d1207b5b13bbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:48:18.689872 31046 raft_consensus.cc:1261] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Refusing update from remote peer 4c93f70b4fe645118f3d1207b5b13bbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:48:18.690354 31392 consensus_queue.cc:1035] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:48:18.690529 31389 consensus_queue.cc:1035] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:48:18.696735 31398 mvcc.cc:205] Tried to move back new op lower bound from 7316041313029287936 to 7316041312945795072. Current Snapshot: MvccSnapshot[applied={T|T < 7316041313029287936}]
I20260807 22:48:18.699433 30899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:48:18.699738 30899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:48:18.701354 30899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: Sending DeleteTablet for 3 replicas of tablet 630d66bbf28f4a958deb4e2e9bf80b8f
I20260807 22:48:18.702076 31161 tablet_service.cc:1514] Processing DeleteTablet for tablet 630d66bbf28f4a958deb4e2e9bf80b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:18 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:58464
I20260807 22:48:18.702067 31026 tablet_service.cc:1514] Processing DeleteTablet for tablet 630d66bbf28f4a958deb4e2e9bf80b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:18 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:55364
I20260807 22:48:18.702108 31298 tablet_service.cc:1514] Processing DeleteTablet for tablet 630d66bbf28f4a958deb4e2e9bf80b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:18 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:48216
I20260807 22:48:18.702394 31406 tablet_replica.cc:330] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db: stopping tablet replica
I20260807 22:48:18.702531 31406 raft_consensus.cc:2229] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:48:18.702597 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 30963
I20260807 22:48:18.702672 31406 raft_consensus.cc:2258] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:48:18.709735 31407 tablet_replica.cc:330] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe: stopping tablet replica
I20260807 22:48:18.709798 31406 ts_tablet_manager.cc:1905] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:48:18.710079 31407 raft_consensus.cc:2229] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:48:18.710289 31407 raft_consensus.cc:2258] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:48:18.711279 31407 ts_tablet_manager.cc:1905] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:48:18.712157 31406 ts_tablet_manager.cc:1918] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:48:18.712265 31406 log.cc:1196] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/630d66bbf28f4a958deb4e2e9bf80b8f
I20260807 22:48:18.712620 31406 ts_tablet_manager.cc:1939] T 630d66bbf28f4a958deb4e2e9bf80b8f P f42693cabf23402eb345705c414c36db: Deleting consensus metadata
I20260807 22:48:18.713469 30885 catalog_manager.cc:4841] TS f42693cabf23402eb345705c414c36db (127.16.98.130:37707): tablet 630d66bbf28f4a958deb4e2e9bf80b8f (table test-table [id=59a444a2af8f44e4a21c6e07a28b718e]) successfully deleted
I20260807 22:48:18.714318 31407 ts_tablet_manager.cc:1918] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:48:18.714413 31407 log.cc:1196] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/630d66bbf28f4a958deb4e2e9bf80b8f
I20260807 22:48:18.714740 31407 ts_tablet_manager.cc:1939] T 630d66bbf28f4a958deb4e2e9bf80b8f P 4c93f70b4fe645118f3d1207b5b13bbe: Deleting consensus metadata
W20260807 22:48:18.715550 30885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:33263 (error 108)
I20260807 22:48:18.715669 30887 catalog_manager.cc:4841] TS 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995): tablet 630d66bbf28f4a958deb4e2e9bf80b8f (table test-table [id=59a444a2af8f44e4a21c6e07a28b718e]) successfully deleted
I20260807 22:48:18.715898 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 31098
I20260807 22:48:18.717396 31380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:18.717265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:33263 (local address 127.0.0.1:55374)
0807 22:48:18.717311 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:18.717363 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:33263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260807 22:48:18.717657 30885 catalog_manager.cc:4584] TS 77016cd86efa44cfb5534027d2120355 (127.16.98.129:33263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 630d66bbf28f4a958deb4e2e9bf80b8f: Network error: Client connection negotiation failed: client connection to 127.16.98.129:33263: connect: Connection refused (error 111)
I20260807 22:48:18.722882 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 31234
I20260807 22:48:18.729120 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 30868
I20260807 22:48:18.737511 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43929
--webserver_interface=127.16.98.190
--webserver_port=32835
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:43929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:18.838189 31409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:18.838436 31409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:18.838462 31409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:18.839663 31409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:48:18.839706 31409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:18.839721 31409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:18.839735 31409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:48:18.839746 31409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:48:18.841418 31409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:43929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=32835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:18.841722 31409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:18.841950 31409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:18.855525 31417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:18.855561 31416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:18.855712 31409 server_base.cc:1034] running on GCE node
W20260807 22:48:18.855551 31419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:18.856750 31409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:18.857197 31409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:18.858420 31409 hybrid_clock.cc:648] HybridClock initialized: now 1786142898858392 us; error 49 us; skew 500 ppm
Aug 07 22:48:18 dist-test-slave-rgb1 krb5kdc[30855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:18.866254 31409 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:48:18.867084 31409 webserver.cc:460] Webserver started at http://127.16.98.190:32835/ using document root <none> and password file <none>
I20260807 22:48:18.867377 31409 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:18.867486 31409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:18.869392 31409 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 22:48:18.870662 31426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:18.871351 31409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:48:18.871513 31409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "c51bff78f1cf4e4d82ff9fb9c632de52"
format_stamp: "Formatted at 2026-08-07 22:48:17 on dist-test-slave-rgb1"
server_key: "6ca1ae94f4683fe7a302a5b5f01f22df"
server_key_iv: "c3a2051eeb2adfa09c1b56e1b6741b26"
server_key_version: "encryptionkey@0"
I20260807 22:48:18.872321 31409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:18.890007 31409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:18.892625 31409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:18.892822 31409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:18.897960 31409 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:43929
I20260807 22:48:18.898236 31479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:43929 every 8 connection(s)
I20260807 22:48:18.898592 31409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:48:18.900233 31480 sys_catalog.cc:263] Verifying existing consensus state
I20260807 22:48:18.900967 31480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: Bootstrap starting.
I20260807 22:48:18.905896 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 31409
I20260807 22:48:18.906273 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:33263
--local_ip_for_outbound_sockets=127.16.98.129
--tserver_master_addrs=127.16.98.190:43929
--webserver_port=46469
--webserver_interface=127.16.98.129
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:48:18.911998 31480 log.cc:824] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:18.915397 31480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:48:18.915886 31480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: Bootstrap complete.
I20260807 22:48:18.920948 31480 raft_consensus.cc:348] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } }
I20260807 22:48:18.921429 31480 raft_consensus.cc:729] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c51bff78f1cf4e4d82ff9fb9c632de52, State: Initialized, Role: FOLLOWER
I20260807 22:48:18.921674 31480 consensus_queue.cc:260] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } }
I20260807 22:48:18.921753 31480 raft_consensus.cc:388] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:48:18.921804 31480 raft_consensus.cc:482] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:48:18.921864 31480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:48:18.923453 31480 raft_consensus.cc:504] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } }
I20260807 22:48:18.923585 31480 leader_election.cc:302] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c51bff78f1cf4e4d82ff9fb9c632de52; no voters:
I20260807 22:48:18.923787 31480 leader_election.cc:288] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 22:48:18.924048 31484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 22:48:18.924121 31480 sys_catalog.cc:564] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:48:18.924449 31484 raft_consensus.cc:686] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 2 LEADER]: Becoming Leader. State: Replica: c51bff78f1cf4e4d82ff9fb9c632de52, State: Running, Role: LEADER
I20260807 22:48:18.924837 31484 consensus_queue.cc:237] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } }
I20260807 22:48:18.925442 31485 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } } }
I20260807 22:48:18.925546 31485 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:18.925894 31486 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c51bff78f1cf4e4d82ff9fb9c632de52. Latest consensus state: current_term: 2 leader_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } } }
I20260807 22:48:18.926002 31486 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:18.926041 31499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:48:18.926939 31499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:48:18.927062 31499 catalog_manager.cc:1259] Loaded cluster ID: be3e6ad8537e4970a8af55a34edb875f
I20260807 22:48:18.927100 31499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:48:18.927362 31499 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260807 22:48:18.932528 31499 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260807 22:48:18.935700 31499 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:48:18.935966 31499 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: Loaded TSK: 0
I20260807 22:48:18.936211 31499 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 22:48:19.021476 31482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:19.021745 31482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:19.021791 31482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:19.023151 31482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:19.023234 31482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:19.023284 31482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:48:19.025626 31482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:33263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=46469
--tserver_master_addrs=127.16.98.190:43929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:19.025965 31482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:19.026288 31482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:19.033314 31508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:19.033670 31507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:19.033994 31482 server_base.cc:1034] running on GCE node
W20260807 22:48:19.034032 31510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:19.034346 31482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:19.034677 31482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:19.035874 31482 hybrid_clock.cc:648] HybridClock initialized: now 1786142899035838 us; error 51 us; skew 500 ppm
Aug 07 22:48:19 dist-test-slave-rgb1 krb5kdc[30855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:19.039595 31482 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:48:19.040225 31482 webserver.cc:460] Webserver started at http://127.16.98.129:46469/ using document root <none> and password file <none>
I20260807 22:48:19.040518 31482 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:19.040668 31482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:19.043028 31482 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:48:19.049163 31517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:19.049628 31482 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260807 22:48:19.049777 31482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "77016cd86efa44cfb5534027d2120355"
format_stamp: "Formatted at 2026-08-07 22:48:18 on dist-test-slave-rgb1"
server_key: "2ba65eb4ec53c414de6f81938b64528f"
server_key_iv: "aa0344ba5453ccd854778deb178b05ce"
server_key_version: "encryptionkey@0"
I20260807 22:48:19.050369 31482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:19.063295 31482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:19.066094 31482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:19.066291 31482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:19.066574 31482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:19.067126 31524 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 22:48:19.069312 31482 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 22:48:19.069394 31482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260807 22:48:19.069439 31482 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 22:48:19.070788 31482 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 22:48:19.070952 31524 tablet_bootstrap.cc:492] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: Bootstrap starting.
I20260807 22:48:19.070990 31482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260807 22:48:19.083675 31482 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:33263
I20260807 22:48:19.083748 31635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:33263 every 8 connection(s)
I20260807 22:48:19.084429 31482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:48:19.085474 31524 log.cc:824] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:19.088475 31524 tablet_bootstrap.cc:492] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:48:19.088930 31524 tablet_bootstrap.cc:492] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: Bootstrap complete.
I20260807 22:48:19.089529 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 31482
I20260807 22:48:19.089938 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:37707
--local_ip_for_outbound_sockets=127.16.98.130
--tserver_master_addrs=127.16.98.190:43929
--webserver_port=42805
--webserver_interface=127.16.98.130
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:48:19.090682 31524 ts_tablet_manager.cc:1397] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.012s
I20260807 22:48:19.092286 31524 raft_consensus.cc:348] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } }
I20260807 22:48:19.092756 31524 raft_consensus.cc:729] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77016cd86efa44cfb5534027d2120355, State: Initialized, Role: FOLLOWER
I20260807 22:48:19.092969 31524 consensus_queue.cc:260] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } }
I20260807 22:48:19.093287 31524 ts_tablet_manager.cc:1428] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Aug 07 22:48:19 dist-test-slave-rgb1 krb5kdc[30855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:19.107220 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.085355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:40115)
0807 22:48:19.085680 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:19.085757 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:19.086439 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:19.093225 (+ 6786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:19.093231 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:19.093643 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:19.093878 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.093885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.095905 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.095907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.097481 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.097486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.098666 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.098668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.098841 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.099574 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:19.099585 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:19.102139 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:19.105098 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:19.105109 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:19.105112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:19.105468 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:19.105808 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:19.105808 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:19.105809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:19.105952 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:19.106463 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:19.106471 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:19.106648 (+ 177us) client_negotiation.cc:769] Sending connection context
0807 22:48:19.106743 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 22:48:19.106940 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":109,"threads_started":1}
I20260807 22:48:19.108461 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.085456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:40115 (local address 127.16.98.190:43929)
0807 22:48:19.092686 (+ 7230us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:19.092699 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:19.092752 (+ 53us) server_negotiation.cc:407] Connection header received
0807 22:48:19.092900 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:19.092909 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:19.092973 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:19.093174 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:19.094034 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.095738 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.097648 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.098511 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.102328 (+ 3817us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.102457 (+ 129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:19.102461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:19.102462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:19.102472 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:19.104957 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:19.105578 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:19.105581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:19.105582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:19.105722 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:19.106050 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:19.106050 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:19.106051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:19.106372 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:19.107939 (+ 1567us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:19.108046 (+ 107us) server_negotiation.cc:299] Negotiation successful
0807 22:48:19.108204 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7164,"thread_start_us":143,"threads_started":1}
I20260807 22:48:19.109688 31636 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43929
I20260807 22:48:19.109824 31636 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:19.110028 31636 heartbeater.cc:511] Master 127.16.98.190:43929 requested a full tablet report, sending...
I20260807 22:48:19.110878 31444 ts_manager.cc:194] Registered new tserver with Master: 77016cd86efa44cfb5534027d2120355 (127.16.98.129:33263)
I20260807 22:48:19.112296 31444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:40115
I20260807 22:48:19.113373 31636 heartbeater.cc:503] Master 127.16.98.190:43929 was elected leader, sending a full tablet report...
Aug 07 22:48:19 dist-test-slave-rgb1 krb5kdc[30855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for kudu/127.16.98.129@KRBTEST.COM
I20260807 22:48:19.123171 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.111621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:33263 (local address 127.0.0.1:55380)
0807 22:48:19.111839 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:19.111875 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:19.111946 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:19.112289 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:19.112296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:19.112829 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:19.112923 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.112926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.114302 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.114304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.115400 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.115404 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.116145 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.116146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.116251 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.116949 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:19.116955 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:19.119045 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:19.121620 (+ 2575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:19.121631 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:19.121633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:19.121940 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:19.122216 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:19.122218 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:19.122218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:19.122286 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:19.122688 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:19.122697 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:19.122761 (+ 64us) client_negotiation.cc:769] Sending connection context
0807 22:48:19.122833 (+ 72us) client_negotiation.cc:240] Negotiation successful
0807 22:48:19.122979 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":151,"threads_started":1}
I20260807 22:48:19.124066 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.111950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.16.98.129:33263)
0807 22:48:19.112117 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:19.112129 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:19.112138 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:48:19.112168 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:19.112169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:19.112181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:19.112272 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:19.113040 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.114188 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.115518 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.116041 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.119153 (+ 3112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.119226 (+ 73us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:19.119231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:19.119231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:19.119237 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:19.121495 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:19.122065 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:19.122068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:19.122069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:19.122138 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:19.122372 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:19.122373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:19.122373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:19.122597 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:19.123739 (+ 1142us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:19.123796 (+ 57us) server_negotiation.cc:299] Negotiation successful
0807 22:48:19.123921 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260807 22:48:19.124898 31568 tablet_service.cc:1514] Processing DeleteTablet for tablet 630d66bbf28f4a958deb4e2e9bf80b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:18 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:55380
I20260807 22:48:19.128733 31645 tablet_replica.cc:330] stopping tablet replica
I20260807 22:48:19.128893 31645 raft_consensus.cc:2229] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:48:19.128993 31645 raft_consensus.cc:2258] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:48:19.131630 31645 ts_tablet_manager.cc:1905] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:48:19.134128 31645 ts_tablet_manager.cc:1918] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:48:19.134248 31645 log.cc:1196] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal/wals/630d66bbf28f4a958deb4e2e9bf80b8f
I20260807 22:48:19.134482 31645 ts_tablet_manager.cc:1939] T 630d66bbf28f4a958deb4e2e9bf80b8f P 77016cd86efa44cfb5534027d2120355: Deleting consensus metadata
I20260807 22:48:19.135389 31428 catalog_manager.cc:4841] TS 77016cd86efa44cfb5534027d2120355 (127.16.98.129:33263): tablet 630d66bbf28f4a958deb4e2e9bf80b8f (table test-table [id=59a444a2af8f44e4a21c6e07a28b718e]) successfully deleted
W20260807 22:48:19.206991 31641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:19.207234 31641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:19.207268 31641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:19.208746 31641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:19.208806 31641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:19.208839 31641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:48:19.210598 31641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:37707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=42805
--tserver_master_addrs=127.16.98.190:43929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:19.210888 31641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:19.211181 31641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:19.217417 31653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:19.217417 31651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:19.217722 31650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:19.217691 31641 server_base.cc:1034] running on GCE node
I20260807 22:48:19.217994 31641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:19.218232 31641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:19.219599 31641 hybrid_clock.cc:648] HybridClock initialized: now 1786142899219572 us; error 45 us; skew 500 ppm
Aug 07 22:48:19 dist-test-slave-rgb1 krb5kdc[30855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:19.222823 31641 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:48:19.223418 31641 webserver.cc:460] Webserver started at http://127.16.98.130:42805/ using document root <none> and password file <none>
I20260807 22:48:19.223691 31641 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:19.223793 31641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:19.225721 31641 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:19.226583 31660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:19.226807 31641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:19.226923 31641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "f42693cabf23402eb345705c414c36db"
format_stamp: "Formatted at 2026-08-07 22:48:18 on dist-test-slave-rgb1"
server_key: "58aacebf8ca45ca010530023af3cb8f0"
server_key_iv: "4658b65b8b7c4ada247583709f1f89b1"
server_key_version: "encryptionkey@0"
I20260807 22:48:19.227378 31641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:19.254469 31641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:19.257302 31641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:19.257489 31641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:19.257762 31641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:19.258116 31641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:19.258176 31641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:19.258221 31641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:19.258253 31641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:19.264917 31641 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:37707
I20260807 22:48:19.265144 31774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:37707 every 8 connection(s)
I20260807 22:48:19.265386 31641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:48:19.272153 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 31641
I20260807 22:48:19.272491 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:39995
--local_ip_for_outbound_sockets=127.16.98.131
--tserver_master_addrs=127.16.98.190:43929
--webserver_port=44363
--webserver_interface=127.16.98.131
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 22:48:19 dist-test-slave-rgb1 krb5kdc[30855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:19.277853 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.266241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:40407 (local address 127.16.98.190:43929)
0807 22:48:19.266312 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:19.266314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:19.266493 (+ 179us) server_negotiation.cc:407] Connection header received
0807 22:48:19.267055 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:19.267056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:19.267067 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:19.267137 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:19.268317 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.269037 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.270312 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.270547 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.270631 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.273449 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:19.273451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:19.273452 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:19.273462 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:19.275860 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:19.276489 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:19.276490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:19.276491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:19.276613 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:19.277041 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:19.277042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:19.277042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:19.277180 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:19.277257 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:19.277686 (+ 429us) server_negotiation.cc:299] Negotiation successful
0807 22:48:19.277779 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 22:48:19.278187 31777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.266191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:40407)
0807 22:48:19.266356 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:19.266401 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:19.266906 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:19.267208 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:19.267211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:19.267993 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:19.268198 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.268203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.269152 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.269153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.270179 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.270182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.270810 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.270811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.270924 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.271624 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:19.271633 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:19.273315 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:19.275981 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:19.275991 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:19.275994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:19.276395 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:19.276698 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:19.276699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:19.276699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:19.276888 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:19.277269 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:19.277277 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:19.277487 (+ 210us) client_negotiation.cc:769] Sending connection context
0807 22:48:19.277652 (+ 165us) client_negotiation.cc:240] Negotiation successful
0807 22:48:19.277879 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260807 22:48:19.279224 31775 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43929
I20260807 22:48:19.279379 31775 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:19.279686 31775 heartbeater.cc:511] Master 127.16.98.190:43929 requested a full tablet report, sending...
I20260807 22:48:19.280309 31444 ts_manager.cc:194] Registered new tserver with Master: f42693cabf23402eb345705c414c36db (127.16.98.130:37707)
I20260807 22:48:19.280907 31444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:40407
W20260807 22:48:19.369526 31778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:19.369854 31778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:19.369886 31778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:19.371213 31778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:19.371264 31778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:19.371294 31778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:48:19.372908 31778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:39995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=44363
--tserver_master_addrs=127.16.98.190:43929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:19.373206 31778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:19.373481 31778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:19.380517 31783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:19.380625 31784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:19.380744 31778 server_base.cc:1034] running on GCE node
W20260807 22:48:19.380625 31786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:19.381057 31778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:19.381371 31778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:19.382594 31778 hybrid_clock.cc:648] HybridClock initialized: now 1786142899382572 us; error 39 us; skew 500 ppm
Aug 07 22:48:19 dist-test-slave-rgb1 krb5kdc[30855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:19.385922 31778 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:48:19.386503 31778 webserver.cc:460] Webserver started at http://127.16.98.131:44363/ using document root <none> and password file <none>
I20260807 22:48:19.386777 31778 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:19.386870 31778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:19.388370 31778 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:48:19.389278 31795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:19.389549 31778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:48:19.389648 31778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "4c93f70b4fe645118f3d1207b5b13bbe"
format_stamp: "Formatted at 2026-08-07 22:48:18 on dist-test-slave-rgb1"
server_key: "451149b16629902ff49a3802c4603e0e"
server_key_iv: "f1b9086ffa1b0c1444f590ba8ba27987"
server_key_version: "encryptionkey@0"
I20260807 22:48:19.390038 31778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:19.415472 31778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:19.418076 31778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:19.418252 31778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:19.418533 31778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:19.418876 31778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:19.418931 31778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:19.419000 31778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:19.419031 31778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:19.426098 31778 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:39995
I20260807 22:48:19.426168 31910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:39995 every 8 connection(s)
I20260807 22:48:19.426559 31778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:48:19.431150 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 31778
Aug 07 22:48:19 dist-test-slave-rgb1 krb5kdc[30855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:19.440851 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.431723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53626 (local address 127.16.98.190:43929)
0807 22:48:19.432021 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:19.432024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:19.432121 (+ 97us) server_negotiation.cc:407] Connection header received
0807 22:48:19.432236 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:19.432237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:19.432251 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:19.432327 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:19.433219 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.433844 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.434948 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.435151 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.435270 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.437150 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:19.437151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:19.437151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:19.437158 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:19.438981 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:19.439539 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:19.439540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:19.439541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:19.439781 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:19.440151 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:19.440152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:19.440152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:19.440267 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:19.440457 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:19.440500 (+ 43us) server_negotiation.cc:299] Negotiation successful
0807 22:48:19.440619 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":89,"threads_started":1}
I20260807 22:48:19.440941 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.427657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:43433 (local address 127.16.98.190:43929)
0807 22:48:19.427713 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:19.427715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:19.427887 (+ 172us) server_negotiation.cc:407] Connection header received
0807 22:48:19.428464 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:19.428466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:19.428480 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:19.428715 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:19.430491 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.431288 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.432668 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.433050 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.433141 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.436214 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:19.436215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:19.436215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:19.436223 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:19.438504 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:19.439161 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:19.439163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:19.439163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:19.439248 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:19.439576 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:19.439577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:19.439577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:19.439961 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:19.440209 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:19.440644 (+ 435us) server_negotiation.cc:299] Negotiation successful
0807 22:48:19.440746 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 22:48:19.441118 31913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.427522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:43433)
0807 22:48:19.427711 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:19.427760 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:19.428309 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:19.428930 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:19.428937 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:19.430031 (+ 1094us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:19.430341 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.430352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.431457 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.431458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.432509 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.432512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.433158 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.433159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.433340 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.434278 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:19.434289 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:19.436026 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:19.438679 (+ 2653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:19.438698 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:19.438701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:19.439054 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:19.439354 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:19.439355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:19.439356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:19.439503 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:19.440202 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:19.440217 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:19.440418 (+ 201us) client_negotiation.cc:769] Sending connection context
0807 22:48:19.440616 (+ 198us) client_negotiation.cc:240] Negotiation successful
0807 22:48:19.440842 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":107,"threads_started":1}
I20260807 22:48:19.441984 31911 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43929
I20260807 22:48:19.442173 31911 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:19.442463 31911 heartbeater.cc:511] Master 127.16.98.190:43929 requested a full tablet report, sending...
I20260807 22:48:19.443169 31444 ts_manager.cc:194] Registered new tserver with Master: 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995)
I20260807 22:48:19.443590 31444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:43433
I20260807 22:48:19.453632 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:48:19.457592 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.454151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.16.98.190:43929)
0807 22:48:19.454251 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:19.454254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:19.454396 (+ 142us) server_negotiation.cc:407] Connection header received
0807 22:48:19.454503 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:19.454504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:19.454568 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:19.454662 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:48:19.454817 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.455458 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.456509 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.456789 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.456860 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.457130 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:48:19.457246 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:48:19.457335 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:19.457488 (+ 153us) server_negotiation.cc:299] Negotiation successful
0807 22:48:19.457519 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 22:48:19.457988 31444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:48:19.459846 31444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:48:19.462728 31568 tablet_service.cc:1467] Processing CreateTablet for tablet e273da03e10949ae8d513e2a5290d9d5 (DEFAULT_TABLE table=test-table [id=61cdfe927b4b4b8697951723a460c068]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:48:19.462940 31568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e273da03e10949ae8d513e2a5290d9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:19.464460 31524 tablet_bootstrap.cc:492] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355: Bootstrap starting.
I20260807 22:48:19.464866 31524 tablet_bootstrap.cc:654] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:19.465873 31524 tablet_bootstrap.cc:492] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355: No bootstrap required, opened a new log
I20260807 22:48:19.465929 31524 ts_tablet_manager.cc:1397] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:19.466099 31524 raft_consensus.cc:348] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:19.466213 31524 raft_consensus.cc:374] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:19.466271 31524 raft_consensus.cc:729] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77016cd86efa44cfb5534027d2120355, State: Initialized, Role: FOLLOWER
I20260807 22:48:19.466317 31524 consensus_queue.cc:260] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:19.466416 31524 ts_tablet_manager.cc:1428] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260807 22:48:19.466838 31920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.462310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:37707 (local address 127.0.0.1:58472)
0807 22:48:19.462415 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:19.462427 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:19.462475 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:19.463203 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:19.463204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:19.463207 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:19.463274 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.463276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.464220 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.464221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.465574 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.465577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.466573 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.466574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.466639 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.466661 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 22:48:19.466712 (+ 51us) client_negotiation.cc:240] Negotiation successful
0807 22:48:19.466757 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":70,"threads_started":1}
I20260807 22:48:19.466917 31922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.462601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58472 (local address 127.16.98.130:37707)
0807 22:48:19.462936 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:19.463009 (+ 73us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:19.463022 (+ 13us) server_negotiation.cc:407] Connection header received
0807 22:48:19.463077 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:19.463079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:19.463093 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:19.463194 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:19.463344 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.464112 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.465667 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.466385 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.466473 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.466536 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:19.466747 (+ 211us) server_negotiation.cc:299] Negotiation successful
0807 22:48:19.466797 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":116,"threads_started":1}
I20260807 22:48:19.467780 31709 tablet_service.cc:1467] Processing CreateTablet for tablet e273da03e10949ae8d513e2a5290d9d5 (DEFAULT_TABLE table=test-table [id=61cdfe927b4b4b8697951723a460c068]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:48:19.468127 31709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e273da03e10949ae8d513e2a5290d9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:19.468454 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.462386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48248 (local address 127.16.98.131:39995)
0807 22:48:19.464515 (+ 2129us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:19.464519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:19.464531 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:48:19.464599 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:19.464601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:19.464618 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:19.464849 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:19.464928 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.465935 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.467289 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.468087 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.468156 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.468214 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:19.468352 (+ 138us) server_negotiation.cc:299] Negotiation successful
0807 22:48:19.468389 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2077,"thread_start_us":164,"threads_started":1}
I20260807 22:48:19.468453 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.462258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:39995 (local address 127.0.0.1:48248)
0807 22:48:19.462322 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:19.462335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:19.462384 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:19.464732 (+ 2348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:19.464734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:19.464736 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:19.464773 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.464774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.466078 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.466080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.467210 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.467213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.468182 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.468183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.468261 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.468266 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:48:19.468340 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 22:48:19.468387 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260807 22:48:19.469195 31843 tablet_service.cc:1467] Processing CreateTablet for tablet e273da03e10949ae8d513e2a5290d9d5 (DEFAULT_TABLE table=test-table [id=61cdfe927b4b4b8697951723a460c068]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:48:19.469511 31843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e273da03e10949ae8d513e2a5290d9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:19.472182 31924 tablet_bootstrap.cc:492] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db: Bootstrap starting.
I20260807 22:48:19.473449 31925 tablet_bootstrap.cc:492] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe: Bootstrap starting.
I20260807 22:48:19.474319 31924 tablet_bootstrap.cc:654] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:19.474668 31925 tablet_bootstrap.cc:654] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:19.475041 31924 log.cc:824] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:19.475338 31925 log.cc:824] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:19.475915 31924 tablet_bootstrap.cc:492] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db: No bootstrap required, opened a new log
I20260807 22:48:19.475987 31924 ts_tablet_manager.cc:1397] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260807 22:48:19.476248 31925 tablet_bootstrap.cc:492] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe: No bootstrap required, opened a new log
I20260807 22:48:19.476334 31925 ts_tablet_manager.cc:1397] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:48:19.478551 31924 raft_consensus.cc:348] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:19.478771 31924 raft_consensus.cc:374] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:19.478822 31924 raft_consensus.cc:729] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42693cabf23402eb345705c414c36db, State: Initialized, Role: FOLLOWER
I20260807 22:48:19.478961 31924 consensus_queue.cc:260] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:19.479012 31925 raft_consensus.cc:348] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:19.479228 31925 raft_consensus.cc:374] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:19.479282 31925 raft_consensus.cc:729] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c93f70b4fe645118f3d1207b5b13bbe, State: Initialized, Role: FOLLOWER
I20260807 22:48:19.479310 31924 ts_tablet_manager.cc:1428] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260807 22:48:19.479367 31775 heartbeater.cc:503] Master 127.16.98.190:43929 was elected leader, sending a full tablet report...
I20260807 22:48:19.479399 31925 consensus_queue.cc:260] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:19.479683 31925 ts_tablet_manager.cc:1428] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260807 22:48:19.479761 31911 heartbeater.cc:503] Master 127.16.98.190:43929 was elected leader, sending a full tablet report...
W20260807 22:48:19.516135 31776 tablet.cc:2369] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:48:19.587162 31637 tablet.cc:2369] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:48:19.629145 31642 raft_consensus.cc:482] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:48:19.629315 31642 raft_consensus.cc:504] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:19.629822 31642 leader_election.cc:288] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995), f42693cabf23402eb345705c414c36db (127.16.98.130:37707)
I20260807 22:48:19.633157 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.630161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:37707 (local address 127.16.98.129:60491)
0807 22:48:19.630316 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:19.630328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:19.630405 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:19.630566 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:19.630567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:19.630570 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:19.630648 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.630650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.631323 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.631324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.632174 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.632176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.632818 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.632819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.632906 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.632925 (+ 19us) client_negotiation.cc:769] Sending connection context
0807 22:48:19.633007 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 22:48:19.633053 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260807 22:48:19.633158 31922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.630211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:60491 (local address 127.16.98.130:37707)
0807 22:48:19.630267 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:19.630270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:19.630417 (+ 147us) server_negotiation.cc:407] Connection header received
0807 22:48:19.630464 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:19.630465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:19.630475 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:19.630544 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:19.630718 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.631200 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.632299 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.632727 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.632819 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.632829 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:19.633016 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 22:48:19.633061 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260807 22:48:19.633539 31729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e273da03e10949ae8d513e2a5290d9d5" candidate_uuid: "77016cd86efa44cfb5534027d2120355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42693cabf23402eb345705c414c36db" is_pre_election: true
I20260807 22:48:19.633618 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.630057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:39995 (local address 127.16.98.129:59675)
0807 22:48:19.630223 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:19.630238 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:19.630392 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:19.630633 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:19.630634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:19.630637 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:19.630702 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.630704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.631728 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.631729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.632658 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:19.632661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.633385 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.633386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:19.633455 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.633460 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:48:19.633518 (+ 58us) client_negotiation.cc:240] Negotiation successful
0807 22:48:19.633557 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260807 22:48:19.633708 31921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.630260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:59675 (local address 127.16.98.131:39995)
0807 22:48:19.630329 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:19.630332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:19.630344 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:48:19.630510 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:19.630512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:19.630524 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:19.630625 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:19.630798 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.631604 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.632761 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.633275 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.633389 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.633401 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:19.633544 (+ 143us) server_negotiation.cc:299] Negotiation successful
0807 22:48:19.633592 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 22:48:19.633720 31729 raft_consensus.cc:2454] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77016cd86efa44cfb5534027d2120355 in term 0.
I20260807 22:48:19.633978 31519 leader_election.cc:302] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77016cd86efa44cfb5534027d2120355, f42693cabf23402eb345705c414c36db; no voters:
I20260807 22:48:19.634018 31864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e273da03e10949ae8d513e2a5290d9d5" candidate_uuid: "77016cd86efa44cfb5534027d2120355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" is_pre_election: true
I20260807 22:48:19.634128 31642 raft_consensus.cc:2781] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:48:19.634173 31864 raft_consensus.cc:2454] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77016cd86efa44cfb5534027d2120355 in term 0.
I20260807 22:48:19.634188 31642 raft_consensus.cc:482] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:48:19.634224 31642 raft_consensus.cc:3037] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:19.635036 31642 raft_consensus.cc:504] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:19.635231 31642 leader_election.cc:288] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 1 election: Requested vote from peers 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995), f42693cabf23402eb345705c414c36db (127.16.98.130:37707)
I20260807 22:48:19.635509 31729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e273da03e10949ae8d513e2a5290d9d5" candidate_uuid: "77016cd86efa44cfb5534027d2120355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42693cabf23402eb345705c414c36db"
I20260807 22:48:19.635535 31864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e273da03e10949ae8d513e2a5290d9d5" candidate_uuid: "77016cd86efa44cfb5534027d2120355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c93f70b4fe645118f3d1207b5b13bbe"
I20260807 22:48:19.635598 31729 raft_consensus.cc:3037] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:19.635648 31864 raft_consensus.cc:3037] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:19.636428 31729 raft_consensus.cc:2454] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77016cd86efa44cfb5534027d2120355 in term 1.
I20260807 22:48:19.636653 31864 raft_consensus.cc:2454] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77016cd86efa44cfb5534027d2120355 in term 1.
I20260807 22:48:19.636735 31519 leader_election.cc:302] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77016cd86efa44cfb5534027d2120355, f42693cabf23402eb345705c414c36db; no voters:
I20260807 22:48:19.636847 31642 raft_consensus.cc:2781] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:48:19.636997 31642 raft_consensus.cc:686] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 LEADER]: Becoming Leader. State: Replica: 77016cd86efa44cfb5534027d2120355, State: Running, Role: LEADER
I20260807 22:48:19.637099 31642 consensus_queue.cc:237] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:19.637925 31444 catalog_manager.cc:5495] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77016cd86efa44cfb5534027d2120355 (127.16.98.129). New cstate: current_term: 1 leader_uuid: "77016cd86efa44cfb5534027d2120355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } health_report { overall_health: HEALTHY } } }
I20260807 22:48:19.656492 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.653767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.16.98.129:33263)
0807 22:48:19.653936 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:19.653938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:19.653945 (+ 7us) server_negotiation.cc:407] Connection header received
0807 22:48:19.653987 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:19.653987 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:19.654010 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:19.654126 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:48:19.654260 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.654790 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.655626 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:19.655840 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:19.655920 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:19.656133 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:48:19.656242 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:48:19.656321 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:19.656366 (+ 45us) server_negotiation.cc:299] Negotiation successful
0807 22:48:19.656390 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260807 22:48:19.658437 31729 raft_consensus.cc:1261] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db [term 1 FOLLOWER]: Refusing update from remote peer 77016cd86efa44cfb5534027d2120355: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:48:19.658643 31864 raft_consensus.cc:1261] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe [term 1 FOLLOWER]: Refusing update from remote peer 77016cd86efa44cfb5534027d2120355: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:48:19.658882 31642 consensus_queue.cc:1035] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:48:19.659104 31933 consensus_queue.cc:1035] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:48:19.662436 31938 mvcc.cc:205] Tried to move back new op lower bound from 7316041316997672960 to 7316041316914171904. Current Snapshot: MvccSnapshot[applied={T|T < 7316041316997672960}]
I20260807 22:48:19.662949 31937 mvcc.cc:205] Tried to move back new op lower bound from 7316041316997672960 to 7316041316914171904. Current Snapshot: MvccSnapshot[applied={T|T < 7316041316997672960}]
I20260807 22:48:19.666486 31443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:48:19.666667 31443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:48:19.668287 31443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: Sending DeleteTablet for 3 replicas of tablet e273da03e10949ae8d513e2a5290d9d5
I20260807 22:48:19.669407 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 31482
I20260807 22:48:19.669360 31568 tablet_service.cc:1514] Processing DeleteTablet for tablet e273da03e10949ae8d513e2a5290d9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:19 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:55380
I20260807 22:48:19.669555 31709 tablet_service.cc:1514] Processing DeleteTablet for tablet e273da03e10949ae8d513e2a5290d9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:19 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:58472
I20260807 22:48:19.670536 31843 tablet_service.cc:1514] Processing DeleteTablet for tablet e273da03e10949ae8d513e2a5290d9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:19 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:48248
I20260807 22:48:19.671253 31948 tablet_replica.cc:330] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe: stopping tablet replica
I20260807 22:48:19.671419 31948 raft_consensus.cc:2229] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:48:19.671561 31948 raft_consensus.cc:2258] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:48:19.672487 31948 ts_tablet_manager.cc:1905] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:48:19.676386 31948 ts_tablet_manager.cc:1918] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:48:19.676532 31948 log.cc:1196] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/e273da03e10949ae8d513e2a5290d9d5
I20260807 22:48:19.676884 31948 ts_tablet_manager.cc:1939] T e273da03e10949ae8d513e2a5290d9d5 P 4c93f70b4fe645118f3d1207b5b13bbe: Deleting consensus metadata
I20260807 22:48:19.677512 31947 tablet_replica.cc:330] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db: stopping tablet replica
I20260807 22:48:19.677811 31947 raft_consensus.cc:2229] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:48:19.678042 31947 raft_consensus.cc:2258] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:48:19.678678 31947 ts_tablet_manager.cc:1905] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:48:19.679319 31430 catalog_manager.cc:4841] TS 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995): tablet e273da03e10949ae8d513e2a5290d9d5 (table test-table [id=61cdfe927b4b4b8697951723a460c068]) successfully deleted
I20260807 22:48:19.681349 31947 ts_tablet_manager.cc:1918] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:48:19.681545 31947 log.cc:1196] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/e273da03e10949ae8d513e2a5290d9d5
W20260807 22:48:19.682034 31428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:33263 (error 108)
I20260807 22:48:19.682289 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 31641
I20260807 22:48:19.682312 31947 ts_tablet_manager.cc:1939] T e273da03e10949ae8d513e2a5290d9d5 P f42693cabf23402eb345705c414c36db: Deleting consensus metadata
I20260807 22:48:19.683517 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.683332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:33263 (local address 127.0.0.1:55412)
0807 22:48:19.683406 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:19.683457 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:33263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260807 22:48:19.683981 31428 catalog_manager.cc:4584] TS 77016cd86efa44cfb5534027d2120355 (127.16.98.129:33263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e273da03e10949ae8d513e2a5290d9d5: Network error: Client connection negotiation failed: client connection to 127.16.98.129:33263: connect: Connection refused (error 111)
I20260807 22:48:19.693749 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 31778
I20260807 22:48:19.694844 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:19.693900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:37707 (local address 127.0.0.1:58480)
0807 22:48:19.694741 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:19.694786 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.130:37707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":813}
I20260807 22:48:19.701532 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 31409
I20260807 22:48:19.711081 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43929
--webserver_interface=127.16.98.190
--webserver_port=32835
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:43929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:19.809331 31950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:19.809561 31950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:19.809587 31950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:19.810895 31950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:48:19.810945 31950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:19.810959 31950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:19.810972 31950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:48:19.810984 31950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:48:19.812758 31950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:43929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=32835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:19.813051 31950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:19.813313 31950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:19.818696 31956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:19.818742 31958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:19.818797 31950 server_base.cc:1034] running on GCE node
W20260807 22:48:19.818670 31955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:19.819352 31950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:19.819729 31950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:19.820935 31950 hybrid_clock.cc:648] HybridClock initialized: now 1786142899820906 us; error 43 us; skew 500 ppm
Aug 07 22:48:19 dist-test-slave-rgb1 krb5kdc[30855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:19.823902 31950 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:48:19.824476 31950 webserver.cc:460] Webserver started at http://127.16.98.190:32835/ using document root <none> and password file <none>
I20260807 22:48:19.824791 31950 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:19.824872 31950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:19.826481 31950 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:48:19.827443 31965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:19.827844 31950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:48:19.828063 31950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "c51bff78f1cf4e4d82ff9fb9c632de52"
format_stamp: "Formatted at 2026-08-07 22:48:17 on dist-test-slave-rgb1"
server_key: "6ca1ae94f4683fe7a302a5b5f01f22df"
server_key_iv: "c3a2051eeb2adfa09c1b56e1b6741b26"
server_key_version: "encryptionkey@0"
I20260807 22:48:19.828789 31950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:19.851094 31950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:19.853967 31950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:19.854173 31950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:19.859910 31950 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:43929
I20260807 22:48:19.859977 32018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:43929 every 8 connection(s)
I20260807 22:48:19.860445 31950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:48:19.862375 32020 sys_catalog.cc:263] Verifying existing consensus state
I20260807 22:48:19.863508 32020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: Bootstrap starting.
I20260807 22:48:19.868772 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 31950
I20260807 22:48:19.869174 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:33263
--local_ip_for_outbound_sockets=127.16.98.129
--tserver_master_addrs=127.16.98.190:43929
--webserver_port=46469
--webserver_interface=127.16.98.129
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:48:19.877369 32020 log.cc:824] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:19.879969 32020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:48:19.880336 32020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52: Bootstrap complete.
I20260807 22:48:19.883158 32020 raft_consensus.cc:348] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } }
I20260807 22:48:19.883424 32020 raft_consensus.cc:729] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c51bff78f1cf4e4d82ff9fb9c632de52, State: Initialized, Role: FOLLOWER
I20260807 22:48:19.883625 32020 consensus_queue.cc:260] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } }
I20260807 22:48:19.883711 32020 raft_consensus.cc:388] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:48:19.883788 32020 raft_consensus.cc:482] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:48:19.883839 32020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 2 FOLLOWER]: Advancing to term 3
I20260807 22:48:19.885187 32020 raft_consensus.cc:504] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } }
I20260807 22:48:19.885291 32020 leader_election.cc:302] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c51bff78f1cf4e4d82ff9fb9c632de52; no voters:
I20260807 22:48:19.885550 32020 leader_election.cc:288] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 22:48:19.885596 32024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 22:48:19.885739 32024 raft_consensus.cc:686] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [term 3 LEADER]: Becoming Leader. State: Replica: c51bff78f1cf4e4d82ff9fb9c632de52, State: Running, Role: LEADER
I20260807 22:48:19.885886 32020 sys_catalog.cc:564] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:48:19.885938 32024 consensus_queue.cc:237] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } }
I20260807 22:48:19.886261 32024 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c51bff78f1cf4e4d82ff9fb9c632de52. Latest consensus state: current_term: 3 leader_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } } }
I20260807 22:48:19.886264 32025 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51bff78f1cf4e4d82ff9fb9c632de52" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43929 } } }
I20260807 22:48:19.886355 32024 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:19.886394 32025 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51bff78f1cf4e4d82ff9fb9c632de52 [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:19.886579 32030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:48:19.887918 32030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:48:19.888092 32030 catalog_manager.cc:1259] Loaded cluster ID: be3e6ad8537e4970a8af55a34edb875f
I20260807 22:48:19.888132 32030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260807 22:48:19.888444 32030 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9090f149ad google::LogMessage::Fail() at ??:0
@ 0x7f9090f1788a google::LogMessage::SendToLog() at ??:0
@ 0x7f9090f144de google::LogMessage::Flush() at ??:0
@ 0x7f9090f15a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9098e350f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9098e3e8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9092de20d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9092dd7954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9093c3b6db start_thread at ??:0
@ 0x7f908fa1261f clone at ??:0
W20260807 22:48:19.978827 32022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:19.979130 32022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:19.979182 32022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:19.980607 32022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:19.980713 32022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:19.980773 32022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:48:19.982611 32022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:33263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=46469
--tserver_master_addrs=127.16.98.190:43929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:19.982964 32022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:19.983240 32022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:19.989116 32056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:19.989306 32022 server_base.cc:1034] running on GCE node
W20260807 22:48:19.989354 32059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:19.989354 32057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:19.989728 32022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:19.990044 32022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:19.991250 32022 hybrid_clock.cc:648] HybridClock initialized: now 1786142899991234 us; error 32 us; skew 500 ppm
Aug 07 22:48:19 dist-test-slave-rgb1 krb5kdc[30855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:19.994707 32022 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:48:19.995275 32022 webserver.cc:460] Webserver started at http://127.16.98.129:46469/ using document root <none> and password file <none>
I20260807 22:48:19.995558 32022 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:19.995641 32022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:19.997368 32022 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:48:19.998279 32066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:19.998498 32022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:19.998602 32022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "77016cd86efa44cfb5534027d2120355"
format_stamp: "Formatted at 2026-08-07 22:48:18 on dist-test-slave-rgb1"
server_key: "2ba65eb4ec53c414de6f81938b64528f"
server_key_iv: "aa0344ba5453ccd854778deb178b05ce"
server_key_version: "encryptionkey@0"
I20260807 22:48:19.999042 32022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:20.032712 32022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:20.035444 32022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:20.035631 32022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:20.035934 32022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:20.036476 32074 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 22:48:20.038609 32022 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 22:48:20.038690 32022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260807 22:48:20.038745 32022 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 22:48:20.039429 32022 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 22:48:20.039490 32022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260807 22:48:20.039562 32074 tablet_bootstrap.cc:492] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355: Bootstrap starting.
I20260807 22:48:20.047573 32022 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:33263
I20260807 22:48:20.047744 32185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:33263 every 8 connection(s)
I20260807 22:48:20.048070 32022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:48:20.048794 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 32022
I20260807 22:48:20.049161 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:37707
--local_ip_for_outbound_sockets=127.16.98.130
--tserver_master_addrs=127.16.98.190:43929
--webserver_port=42805
--webserver_interface=127.16.98.130
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:48:20.051718 32074 log.cc:824] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:20.054986 32074 tablet_bootstrap.cc:492] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:48:20.055392 32074 tablet_bootstrap.cc:492] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355: Bootstrap complete.
I20260807 22:48:20.056886 32074 ts_tablet_manager.cc:1397] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.009s
I20260807 22:48:20.058583 32074 raft_consensus.cc:348] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:20.058952 32074 raft_consensus.cc:729] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77016cd86efa44cfb5534027d2120355, State: Initialized, Role: FOLLOWER
I20260807 22:48:20.059231 32074 consensus_queue.cc:260] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:20.059612 32074 ts_tablet_manager.cc:1428] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20260807 22:48:20.119642 32188 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:48:20.048971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:51763)
0807 22:48:20.049468 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:20.049528 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:20.050414 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:20.119403 (+ 68989us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":132,"threads_started":1}
W20260807 22:48:20.119936 32186 heartbeater.cc:644] Failed to heartbeat to 127.16.98.190:43929 (0 consecutive failures): Network error: Failed to ping master at 127.16.98.190:43929: Client connection negotiation failed: client connection to 127.16.98.190:43929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 22:48:20.120397 32188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:20.120265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:59061)
0807 22:48:20.120325 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:20.120372 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260807 22:48:20.120808 32188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:20.120714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:33815)
0807 22:48:20.120761 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:20.120789 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260807 22:48:20.120959 32186 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 22:48:20.146577 32189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:20.146818 32189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:20.146847 32189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:20.148262 32189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:20.148325 32189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:20.148365 32189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:48:20.150108 32189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:37707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=42805
--tserver_master_addrs=127.16.98.190:43929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:20.150342 32189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:20.150787 32189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:20.156548 32197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:20.156600 32189 server_base.cc:1034] running on GCE node
W20260807 22:48:20.156548 32199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:20.156875 32196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:20.157150 32189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:20.157436 32189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:20.158631 32189 hybrid_clock.cc:648] HybridClock initialized: now 1786142900158614 us; error 37 us; skew 500 ppm
Aug 07 22:48:20 dist-test-slave-rgb1 krb5kdc[30855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142900, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:20.161823 32189 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:48:20.162482 32189 webserver.cc:460] Webserver started at http://127.16.98.130:42805/ using document root <none> and password file <none>
I20260807 22:48:20.162770 32189 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:20.162851 32189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:20.164688 32189 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 22:48:20.165688 32207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:20.166149 32189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:48:20.166487 32189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "f42693cabf23402eb345705c414c36db"
format_stamp: "Formatted at 2026-08-07 22:48:18 on dist-test-slave-rgb1"
server_key: "58aacebf8ca45ca010530023af3cb8f0"
server_key_iv: "4658b65b8b7c4ada247583709f1f89b1"
server_key_version: "encryptionkey@0"
I20260807 22:48:20.166975 32189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:20.185153 32189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:20.187567 32189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:20.187774 32189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:20.188112 32189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:20.188478 32189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:20.188540 32189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:20.188627 32189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:20.188675 32189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:20.195892 32189 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:37707
I20260807 22:48:20.195955 32323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:37707 every 8 connection(s)
I20260807 22:48:20.196334 32189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:48:20.197294 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 32189
I20260807 22:48:20.197630 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:39995
--local_ip_for_outbound_sockets=127.16.98.131
--tserver_master_addrs=127.16.98.190:43929
--webserver_port=44363
--webserver_interface=127.16.98.131
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:48:20.197952 32326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:20.197369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:45799)
0807 22:48:20.197610 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:20.197708 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":120,"threads_started":1}
W20260807 22:48:20.198391 32324 heartbeater.cc:644] Failed to heartbeat to 127.16.98.190:43929 (0 consecutive failures): Network error: Failed to ping master at 127.16.98.190:43929: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
I20260807 22:48:20.198735 32326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:20.198632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:44563)
0807 22:48:20.198685 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:20.198712 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260807 22:48:20.199030 32326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:20.198934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:48487)
0807 22:48:20.198987 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:20.199010 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260807 22:48:20.199142 32324 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 22:48:20.289137 32327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:20.289434 32327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:20.289484 32327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:20.290859 32327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:20.290946 32327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:20.291018 32327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:48:20.292924 32327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:39995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=44363
--tserver_master_addrs=127.16.98.190:43929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:20.293257 32327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:20.293541 32327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:20.299899 32334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:20.299947 32336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:20.300195 32327 server_base.cc:1034] running on GCE node
W20260807 22:48:20.299918 32333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:20.300539 32327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:20.300848 32327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:20.302062 32327 hybrid_clock.cc:648] HybridClock initialized: now 1786142900302036 us; error 55 us; skew 500 ppm
Aug 07 22:48:20 dist-test-slave-rgb1 krb5kdc[30855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142900, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:20.305023 32327 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:48:20.305583 32327 webserver.cc:460] Webserver started at http://127.16.98.131:44363/ using document root <none> and password file <none>
I20260807 22:48:20.305836 32327 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:20.305891 32327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:20.307721 32327 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260807 22:48:20.308938 32343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:20.309387 32327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:20.309522 32327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "4c93f70b4fe645118f3d1207b5b13bbe"
format_stamp: "Formatted at 2026-08-07 22:48:18 on dist-test-slave-rgb1"
server_key: "451149b16629902ff49a3802c4603e0e"
server_key_iv: "f1b9086ffa1b0c1444f590ba8ba27987"
server_key_version: "encryptionkey@0"
I20260807 22:48:20.310034 32327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:20.325883 32327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:20.328413 32327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:20.328689 32327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:20.328963 32327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:20.329301 32327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:20.329356 32327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:20.329399 32327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:20.329439 32327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:20.336093 32327 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:39995
I20260807 22:48:20.336136 32459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:39995 every 8 connection(s)
I20260807 22:48:20.336506 32327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:48:20.337738 32462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:20.337234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:59861)
0807 22:48:20.337495 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:20.337588 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
W20260807 22:48:20.338136 32460 heartbeater.cc:644] Failed to heartbeat to 127.16.98.190:43929 (0 consecutive failures): Network error: Failed to ping master at 127.16.98.190:43929: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
I20260807 22:48:20.338476 32462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:20.338355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:50553)
0807 22:48:20.338417 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:20.338457 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260807 22:48:20.338773 32462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:20.338721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:60139)
0807 22:48:20.338744 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:20.338762 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260807 22:48:20.338884 32460 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 22:48:20.344832 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 32327
I20260807 22:48:21.121757 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:21.121468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:55959)
0807 22:48:21.121628 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:21.121688 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260807 22:48:21.199802 32465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:21.199551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:33773)
0807 22:48:21.199718 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:21.199758 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260807 22:48:21.339563 32466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:21.339280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:42889)
0807 22:48:21.339457 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:21.339508 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260807 22:48:21.642428 32467 raft_consensus.cc:482] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:48:21.642643 32467 raft_consensus.cc:504] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:21.643097 32467 leader_election.cc:288] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995), f42693cabf23402eb345705c414c36db (127.16.98.130:37707)
Aug 07 22:48:21 dist-test-slave-rgb1 krb5kdc[30855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.130@KRBTEST.COM
Aug 07 22:48:21 dist-test-slave-rgb1 krb5kdc[30855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.131@KRBTEST.COM
I20260807 22:48:21.654486 32470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:21.643575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:33283 (local address 127.16.98.131:39995)
0807 22:48:21.643737 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:21.643748 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:21.643777 (+ 29us) server_negotiation.cc:407] Connection header received
0807 22:48:21.644226 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:21.644228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:21.644270 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:21.644387 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:21.645290 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:21.646165 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:21.647249 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:21.647665 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:21.647786 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:21.651012 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:21.651016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:21.651017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:21.651029 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:21.652834 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:21.653262 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:21.653265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:21.653265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:21.653363 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:21.653617 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:21.653618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:21.653618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:21.653853 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:21.653944 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:21.654214 (+ 270us) server_negotiation.cc:299] Negotiation successful
0807 22:48:21.654353 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260807 22:48:21.654474 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:21.643337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:39995 (local address 127.16.98.129:33283)
0807 22:48:21.643471 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:21.643484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:21.643550 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:21.644488 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:21.644491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:21.644977 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:21.645147 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:21.645154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:21.646280 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:21.646281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:21.647126 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:21.647129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:21.647773 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:21.647774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:21.647911 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:21.648499 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:21.648528 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:21.650866 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:21.652926 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:21.652928 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:21.652928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:21.653171 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:21.653440 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:21.653441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:21.653441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:21.653539 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:21.653938 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:21.653939 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:21.654045 (+ 106us) client_negotiation.cc:769] Sending connection context
0807 22:48:21.654153 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 22:48:21.654366 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":101,"thread_start_us":54,"threads_started":1}
I20260807 22:48:21.654506 32471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:21.643617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:32849 (local address 127.16.98.130:37707)
0807 22:48:21.643770 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:21.643786 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:21.643806 (+ 20us) server_negotiation.cc:407] Connection header received
0807 22:48:21.644276 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:21.644278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:21.644317 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:21.644514 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:21.645296 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:21.646238 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:21.647216 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:21.647721 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:21.648099 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:21.650572 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:21.650577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:21.650577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:21.650594 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:21.652663 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:21.653172 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:21.653175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:21.653176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:21.653282 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:21.653547 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:21.653548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:21.653548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:21.653817 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:21.653910 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:21.654220 (+ 310us) server_negotiation.cc:299] Negotiation successful
0807 22:48:21.654359 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":57,"threads_started":1}
I20260807 22:48:21.655247 32468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:21.643337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:37707 (local address 127.16.98.129:32849)
0807 22:48:21.643472 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:21.643484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:21.643545 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:21.644519 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:21.644520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:21.644941 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:21.645147 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:21.645154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:21.646349 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:21.646350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:21.647120 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:21.647123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:21.648046 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:21.648047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:21.648106 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:21.648706 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:21.648711 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:21.650435 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:21.652769 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:21.652779 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:21.652781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:21.653080 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:21.653361 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:21.653362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:21.653362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:21.653474 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:21.653901 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:21.653908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:21.654047 (+ 139us) client_negotiation.cc:769] Sending connection context
0807 22:48:21.654152 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 22:48:21.654452 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":119,"thread_start_us":90,"threads_started":1}
W20260807 22:48:21.655550 32071 leader_election.cc:341] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995): Not found: Tablet not found: e273da03e10949ae8d513e2a5290d9d5
W20260807 22:48:21.656114 32068 leader_election.cc:341] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f42693cabf23402eb345705c414c36db (127.16.98.130:37707): Not found: Tablet not found: e273da03e10949ae8d513e2a5290d9d5
I20260807 22:48:21.656247 32068 leader_election.cc:302] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77016cd86efa44cfb5534027d2120355; no voters: 4c93f70b4fe645118f3d1207b5b13bbe, f42693cabf23402eb345705c414c36db
I20260807 22:48:21.656468 32467 raft_consensus.cc:2726] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 22:48:22.122658 32468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:22.122469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:48807)
0807 22:48:22.122564 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:22.122615 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 22:48:22.200817 32472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:22.200469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:60481)
0807 22:48:22.200726 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:22.200771 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":150,"threads_started":1}
I20260807 22:48:22.340360 32473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:22.340119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:47147)
0807 22:48:22.340266 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:22.340309 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260807 22:48:23.123667 32474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:23.123268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:49005)
0807 22:48:23.123584 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:23.123618 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":250,"threads_started":1}
I20260807 22:48:23.201680 32475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:23.201425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:38363)
0807 22:48:23.201586 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:23.201632 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260807 22:48:23.341208 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:23.340973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:33319)
0807 22:48:23.341121 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:23.341163 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260807 22:48:23.420290 32478 raft_consensus.cc:482] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:48:23.420440 32478 raft_consensus.cc:504] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:23.420734 32478 leader_election.cc:288] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995), f42693cabf23402eb345705c414c36db (127.16.98.130:37707)
W20260807 22:48:23.421411 32068 leader_election.cc:341] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f42693cabf23402eb345705c414c36db (127.16.98.130:37707): Not found: Tablet not found: e273da03e10949ae8d513e2a5290d9d5
W20260807 22:48:23.421681 32071 leader_election.cc:341] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995): Not found: Tablet not found: e273da03e10949ae8d513e2a5290d9d5
I20260807 22:48:23.421808 32071 leader_election.cc:302] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77016cd86efa44cfb5534027d2120355; no voters: 4c93f70b4fe645118f3d1207b5b13bbe, f42693cabf23402eb345705c414c36db
I20260807 22:48:23.421932 32478 raft_consensus.cc:2726] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 22:48:24.124707 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:24.124336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:40325)
0807 22:48:24.124541 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:24.124655 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":84,"threads_started":1}
I20260807 22:48:24.202550 32480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:24.202291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:59613)
0807 22:48:24.202442 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:24.202495 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
I20260807 22:48:24.341923 32482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:24.341674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:35441)
0807 22:48:24.341832 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:24.341875 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
I20260807 22:48:25.125689 32483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:25.125424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:49021)
0807 22:48:25.125593 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:25.125639 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":76,"threads_started":1}
I20260807 22:48:25.203766 32484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:25.203327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:49823)
0807 22:48:25.203663 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:25.203714 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":90,"threads_started":1}
I20260807 22:48:25.342968 32485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:25.342659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:60899)
0807 22:48:25.342867 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:25.342915 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
I20260807 22:48:26.126850 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:26.126577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:55389)
0807 22:48:26.126744 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:26.126789 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260807 22:48:26.191092 32487 raft_consensus.cc:482] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:48:26.191200 32487 raft_consensus.cc:504] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:26.191514 32487 leader_election.cc:288] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995), f42693cabf23402eb345705c414c36db (127.16.98.130:37707)
W20260807 22:48:26.191970 32068 leader_election.cc:341] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f42693cabf23402eb345705c414c36db (127.16.98.130:37707): Not found: Tablet not found: e273da03e10949ae8d513e2a5290d9d5
W20260807 22:48:26.192058 32071 leader_election.cc:341] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995): Not found: Tablet not found: e273da03e10949ae8d513e2a5290d9d5
I20260807 22:48:26.192098 32068 leader_election.cc:302] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77016cd86efa44cfb5534027d2120355; no voters: 4c93f70b4fe645118f3d1207b5b13bbe, f42693cabf23402eb345705c414c36db
I20260807 22:48:26.192173 32487 raft_consensus.cc:2726] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 22:48:26.204768 32488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:26.204445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:45757)
0807 22:48:26.204681 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:26.204719 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":172,"threads_started":1}
I20260807 22:48:26.343830 32489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:26.343576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:47889)
0807 22:48:26.343739 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:26.343781 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260807 22:48:27.127830 32490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:27.127504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:42417)
0807 22:48:27.127719 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:27.127768 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":110,"threads_started":1}
I20260807 22:48:27.205554 32491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:27.205284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:47081)
0807 22:48:27.205449 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:27.205495 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260807 22:48:27.344920 32492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:27.344522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:53681)
0807 22:48:27.344734 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:27.344788 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":124,"threads_started":1}
I20260807 22:48:27.922732 32493 raft_consensus.cc:482] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:48:27.922856 32493 raft_consensus.cc:504] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:27.923126 32493 leader_election.cc:288] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995), f42693cabf23402eb345705c414c36db (127.16.98.130:37707)
W20260807 22:48:27.923609 32068 leader_election.cc:341] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f42693cabf23402eb345705c414c36db (127.16.98.130:37707): Not found: Tablet not found: e273da03e10949ae8d513e2a5290d9d5
W20260807 22:48:27.923719 32071 leader_election.cc:341] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995): Not found: Tablet not found: e273da03e10949ae8d513e2a5290d9d5
I20260807 22:48:27.923777 32071 leader_election.cc:302] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77016cd86efa44cfb5534027d2120355; no voters: 4c93f70b4fe645118f3d1207b5b13bbe, f42693cabf23402eb345705c414c36db
I20260807 22:48:27.923936 32493 raft_consensus.cc:2726] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 22:48:28.128963 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:28.128708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:44191)
0807 22:48:28.128859 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:28.128903 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
I20260807 22:48:28.206432 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:28.206172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:55787)
0807 22:48:28.206336 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:28.206383 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":69,"threads_started":1}
I20260807 22:48:28.345986 32496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:28.345696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:58065)
0807 22:48:28.345846 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:28.345928 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260807 22:48:29.129833 32497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:29.129584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:44287)
0807 22:48:29.129735 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:29.129782 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260807 22:48:29.207538 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:29.207197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:50721)
0807 22:48:29.207410 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:29.207484 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260807 22:48:29.346822 32499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:29.346540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:40983)
0807 22:48:29.346723 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:29.346773 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":91,"threads_started":1}
I20260807 22:48:30.130910 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:30.130576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:47785)
0807 22:48:30.130804 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:30.130855 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":95,"threads_started":1}
I20260807 22:48:30.208431 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:30.208163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:51373)
0807 22:48:30.208335 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:30.208384 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260807 22:48:30.347754 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:30.347445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:48115)
0807 22:48:30.347638 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:30.347699 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
I20260807 22:48:31.131867 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:31.131570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:44325)
0807 22:48:31.131762 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:31.131814 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":76,"threads_started":1}
I20260807 22:48:31.209612 32504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:31.209209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:34567)
0807 22:48:31.209458 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:31.209539 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":73,"threads_started":1}
I20260807 22:48:31.348793 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:31.348440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:40069)
0807 22:48:31.348692 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:31.348738 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
I20260807 22:48:32.132884 32506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:32.132544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:42339)
0807 22:48:32.132786 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:32.132831 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":75,"threads_started":1}
I20260807 22:48:32.210594 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:32.210295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:46381)
0807 22:48:32.210497 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:32.210542 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":108,"threads_started":1}
I20260807 22:48:32.349614 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:32.349302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:58851)
0807 22:48:32.349507 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:32.349561 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
I20260807 22:48:33.133747 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:33.133482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:57077)
0807 22:48:33.133648 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:33.133696 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260807 22:48:33.211515 32510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:33.211203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:34411)
0807 22:48:33.211412 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:33.211465 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
I20260807 22:48:33.350526 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:33.350263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:41171)
0807 22:48:33.350428 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:33.350474 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":70,"threads_started":1}
I20260807 22:48:34.134758 32512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:34.134486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:44815)
0807 22:48:34.134658 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:34.134706 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
I20260807 22:48:34.181202 32513 raft_consensus.cc:482] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:48:34.181324 32513 raft_consensus.cc:504] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c93f70b4fe645118f3d1207b5b13bbe" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 39995 } } peers { permanent_uuid: "f42693cabf23402eb345705c414c36db" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 37707 } } peers { permanent_uuid: "77016cd86efa44cfb5534027d2120355" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33263 } }
I20260807 22:48:34.181535 32513 leader_election.cc:288] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995), f42693cabf23402eb345705c414c36db (127.16.98.130:37707)
W20260807 22:48:34.182080 32071 leader_election.cc:341] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c93f70b4fe645118f3d1207b5b13bbe (127.16.98.131:39995): Not found: Tablet not found: e273da03e10949ae8d513e2a5290d9d5
W20260807 22:48:34.182199 32068 leader_election.cc:341] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f42693cabf23402eb345705c414c36db (127.16.98.130:37707): Not found: Tablet not found: e273da03e10949ae8d513e2a5290d9d5
I20260807 22:48:34.182271 32068 leader_election.cc:302] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77016cd86efa44cfb5534027d2120355; no voters: 4c93f70b4fe645118f3d1207b5b13bbe, f42693cabf23402eb345705c414c36db
I20260807 22:48:34.182391 32513 raft_consensus.cc:2726] T e273da03e10949ae8d513e2a5290d9d5 P 77016cd86efa44cfb5534027d2120355 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 22:48:34.212697 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:34.212309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:45457)
0807 22:48:34.212487 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:34.212537 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":82,"threads_started":1}
I20260807 22:48:34.351384 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:34.351120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:45565)
0807 22:48:34.351276 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:34.351321 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260807 22:48:35.135771 32516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:35.135468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.129:58859)
0807 22:48:35.135628 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:35.135703 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260807 22:48:35.213701 32517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:35.213433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.130:33225)
0807 22:48:35.213610 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:35.213653 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":97,"threads_started":1}
I20260807 22:48:35.349692 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 32022
I20260807 22:48:35.352425 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:35.352171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43929 (local address 127.16.98.131:49079)
0807 22:48:35.352309 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:35.352360 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260807 22:48:35.356823 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 32189
I20260807 22:48:35.364518 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 32327
2026-08-07T22:48:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18956 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:48:35 dist-test-slave-rgb1 krb5kdc[32521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:48:35 dist-test-slave-rgb1 krb5kdc[32521](info): set up 2 sockets
Aug 07 22:48:35 dist-test-slave-rgb1 krb5kdc[32521](info): commencing operation
krb5kdc: starting...
W20260807 22:48:36.501194 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:48:36 dist-test-slave-rgb1 krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:48:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:48:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:48:36.644743 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43079
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:43079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:36.731273 32534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:36.731496 32534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:36.731521 32534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:36.732776 32534 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 22:48:36.732820 32534 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 22:48:36.732832 32534 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 22:48:36.732843 32534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:48:36.732857 32534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:36.732869 32534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:36.732913 32534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:48:36.732930 32534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:48:36.734452 32534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:43079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:36.734714 32534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:36.734964 32534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:36.741003 32542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:36.741035 32540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:36.741175 32534 server_base.cc:1034] running on GCE node
W20260807 22:48:36.741151 32539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:36.741597 32534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:36.741891 32534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:36.743075 32534 hybrid_clock.cc:648] HybridClock initialized: now 1786142916743054 us; error 43 us; skew 500 ppm
Aug 07 22:48:36 dist-test-slave-rgb1 krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142916, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:36.745798 32534 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:48:36.746249 32534 webserver.cc:460] Webserver started at http://127.16.98.190:36629/ using document root <none> and password file <none>
I20260807 22:48:36.746520 32534 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:36.746593 32534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:36.746783 32534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:48:36.747900 32534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "86e3a23ef2354b1d89292ed4831f40c0"
format_stamp: "Formatted at 2026-08-07 22:48:36 on dist-test-slave-rgb1"
server_key: "973af8f4f6a612c2f77c6d796eec8064"
server_key_iv: "ba523c21184d363110ffd5d0b6f22d14"
server_key_version: "encryptionkey@0"
I20260807 22:48:36.748275 32534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "86e3a23ef2354b1d89292ed4831f40c0"
format_stamp: "Formatted at 2026-08-07 22:48:36 on dist-test-slave-rgb1"
server_key: "973af8f4f6a612c2f77c6d796eec8064"
server_key_iv: "ba523c21184d363110ffd5d0b6f22d14"
server_key_version: "encryptionkey@0"
I20260807 22:48:36.749835 32534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:48:36.750736 32549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:36.751071 32534 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:36.751190 32534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "86e3a23ef2354b1d89292ed4831f40c0"
format_stamp: "Formatted at 2026-08-07 22:48:36 on dist-test-slave-rgb1"
server_key: "973af8f4f6a612c2f77c6d796eec8064"
server_key_iv: "ba523c21184d363110ffd5d0b6f22d14"
server_key_version: "encryptionkey@0"
I20260807 22:48:36.751276 32534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:36.759590 32534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:36.762138 32534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:36.762308 32534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:36.766973 32534 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:43079
I20260807 22:48:36.767122 32603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:43079 every 8 connection(s)
I20260807 22:48:36.767423 32534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:48:36.768400 32604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:36.771129 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 32534
I20260807 22:48:36.771219 32604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Bootstrap starting.
I20260807 22:48:36.771257 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:48:36.771477 16778 external_mini_cluster.cc:1428] Setting key bd10d2dedc8c38e8dd56475344c6aa4e
I20260807 22:48:36.772127 32604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:36.772485 32604 log.cc:824] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:36.773332 32604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: No bootstrap required, opened a new log
I20260807 22:48:36.775643 32604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } }
I20260807 22:48:36.775828 32604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:36.775866 32604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e3a23ef2354b1d89292ed4831f40c0, State: Initialized, Role: FOLLOWER
I20260807 22:48:36.776021 32604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } }
I20260807 22:48:36.776136 32604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:48:36.776185 32604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:48:36.776234 32604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:36.777104 32604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } }
I20260807 22:48:36.777240 32604 leader_election.cc:302] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86e3a23ef2354b1d89292ed4831f40c0; no voters:
I20260807 22:48:36.777489 32604 leader_election.cc:288] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:48:36.777596 32609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:48:36.777810 32604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:48:36.777839 32609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 1 LEADER]: Becoming Leader. State: Replica: 86e3a23ef2354b1d89292ed4831f40c0, State: Running, Role: LEADER
I20260807 22:48:36.778069 32609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } }
I20260807 22:48:36.778445 32609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86e3a23ef2354b1d89292ed4831f40c0. Latest consensus state: current_term: 1 leader_uuid: "86e3a23ef2354b1d89292ed4831f40c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } } }
I20260807 22:48:36.778452 32611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86e3a23ef2354b1d89292ed4831f40c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } } }
I20260807 22:48:36.778620 32609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:36.778630 32611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [sys.catalog]: This master's current role is: LEADER
Aug 07 22:48:36 dist-test-slave-rgb1 krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:36.779593 32618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:48:36.780151 32618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:48:36.782310 32618 catalog_manager.cc:1347] Generated new cluster ID: 75bcb32455bf41f49b13950f9d8a3352
I20260807 22:48:36.782510 32618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:48:36.784855 32607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:36.772140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.16.98.190:43079)
0807 22:48:36.772381 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:36.772397 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:36.772424 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:48:36.772925 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:36.772932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:36.772981 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:36.773099 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:36.773976 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:36.774899 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:36.776003 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:36.776621 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:36.776734 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:36.779997 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:36.780002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:36.780003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:36.780017 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:36.782413 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:36.783014 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:36.783018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:36.783019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:36.783141 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:36.783382 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:36.783383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:36.783383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:36.783756 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:36.783890 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:36.784392 (+ 502us) server_negotiation.cc:299] Negotiation successful
0807 22:48:36.784590 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":128,"threads_started":1}
I20260807 22:48:36.796649 32618 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:48:36.797287 32618 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:48:36.802819 32618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Generated new TSK 0
I20260807 22:48:36.803058 32618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:48:36.837333 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43079
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:36.933053 32631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:36.933290 32631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:36.933319 32631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:36.934643 32631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:36.934695 32631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:36.934726 32631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:48:36.936548 32631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:36.936905 32631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:36.937217 32631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:36.943111 32636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:36.943140 32637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:36.943111 32639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:36.943508 32631 server_base.cc:1034] running on GCE node
I20260807 22:48:36.943765 32631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:36.944207 32631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:36.945416 32631 hybrid_clock.cc:648] HybridClock initialized: now 1786142916945384 us; error 87 us; skew 500 ppm
Aug 07 22:48:36 dist-test-slave-rgb1 krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142916, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:36.948531 32631 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:48:36.949205 32631 webserver.cc:460] Webserver started at http://127.16.98.129:33585/ using document root <none> and password file <none>
I20260807 22:48:36.949486 32631 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:36.949559 32631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:36.949715 32631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:48:36.950847 32631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "705e830ad7b74ed6b85e48ec392fb19a"
format_stamp: "Formatted at 2026-08-07 22:48:36 on dist-test-slave-rgb1"
server_key: "b3efa980fa077a4db385ac0a9acc4433"
server_key_iv: "24b1fdc10c2bd1c3b6895d33103976ce"
server_key_version: "encryptionkey@0"
I20260807 22:48:36.951237 32631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "705e830ad7b74ed6b85e48ec392fb19a"
format_stamp: "Formatted at 2026-08-07 22:48:36 on dist-test-slave-rgb1"
server_key: "b3efa980fa077a4db385ac0a9acc4433"
server_key_iv: "24b1fdc10c2bd1c3b6895d33103976ce"
server_key_version: "encryptionkey@0"
I20260807 22:48:36.953164 32631 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260807 22:48:36.954123 32646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:36.954340 32631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:36.954455 32631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "705e830ad7b74ed6b85e48ec392fb19a"
format_stamp: "Formatted at 2026-08-07 22:48:36 on dist-test-slave-rgb1"
server_key: "b3efa980fa077a4db385ac0a9acc4433"
server_key_iv: "24b1fdc10c2bd1c3b6895d33103976ce"
server_key_version: "encryptionkey@0"
I20260807 22:48:36.954586 32631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:36.976661 32631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:36.979704 32631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:36.979898 32631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:36.980209 32631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:36.980651 32631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:36.980713 32631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:36.980759 32631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:36.980791 32631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:36.988376 32631 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:33245
I20260807 22:48:36.988463 32761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:33245 every 8 connection(s)
I20260807 22:48:36.988880 32631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:48:36.995167 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 32631
I20260807 22:48:36.995309 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:48:36.995565 16778 external_mini_cluster.cc:1428] Setting key 99c583aad02d506799af8620b0e66e19
Aug 07 22:48:36 dist-test-slave-rgb1 krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142916, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:37.001143 32607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:36.989778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:34813 (local address 127.16.98.190:43079)
0807 22:48:36.989911 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:36.989915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:36.990116 (+ 201us) server_negotiation.cc:407] Connection header received
0807 22:48:36.990737 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:36.990738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:36.990753 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:36.990819 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:36.991933 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:36.992504 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:36.993748 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:36.993993 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:36.994092 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:36.997176 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:36.997178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:36.997179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:36.997188 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:36.999107 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:36.999629 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:36.999630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:36.999631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:36.999697 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:37.000059 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:37.000060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:37.000061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:37.000306 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:37.000389 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:37.000978 (+ 589us) server_negotiation.cc:299] Negotiation successful
0807 22:48:37.001071 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 22:48:37.001556 32765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:36.989709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:34813)
0807 22:48:36.989979 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:36.990019 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:36.990607 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:36.990911 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:36.990915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:36.991587 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:36.991804 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:36.991809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:36.992696 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:36.992697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:36.993642 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:36.993645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:36.994112 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:36.994113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:36.994254 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:36.995013 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:36.995025 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:36.997068 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:36.999216 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:36.999228 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:36.999232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:36.999526 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:36.999776 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:36.999777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:36.999778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:36.999947 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:37.000407 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:37.000416 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:37.000748 (+ 332us) client_negotiation.cc:769] Sending connection context
0807 22:48:37.000891 (+ 143us) client_negotiation.cc:240] Negotiation successful
0807 22:48:37.001229 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":103,"threads_started":1}
I20260807 22:48:37.002538 32762 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43079
I20260807 22:48:37.002686 32762 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:37.002938 32762 heartbeater.cc:511] Master 127.16.98.190:43079 requested a full tablet report, sending...
I20260807 22:48:37.003659 32566 ts_manager.cc:194] Registered new tserver with Master: 705e830ad7b74ed6b85e48ec392fb19a (127.16.98.129:33245)
I20260807 22:48:37.004876 32566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:34813
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:48:37.029934 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43079
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:37.121946 301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:37.122190 301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:37.122217 301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:37.123457 301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:37.123507 301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:37.123538 301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:48:37.125144 301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:37.125411 301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:37.125694 301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:37.130995 306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:37.131058 307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:37.131050 309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:37.131240 301 server_base.cc:1034] running on GCE node
I20260807 22:48:37.131584 301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:37.131843 301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:37.133085 301 hybrid_clock.cc:648] HybridClock initialized: now 1786142917133098 us; error 61 us; skew 500 ppm
Aug 07 22:48:37 dist-test-slave-rgb1 krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142917, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:37.136099 301 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:48:37.136678 301 webserver.cc:460] Webserver started at http://127.16.98.130:43485/ using document root <none> and password file <none>
I20260807 22:48:37.136909 301 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:37.136960 301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:37.137059 301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:48:37.138131 301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "4df16ca22bc248069ab8864eeab8d81a"
format_stamp: "Formatted at 2026-08-07 22:48:37 on dist-test-slave-rgb1"
server_key: "f432dc09089422c0bddf2d447d9501f9"
server_key_iv: "9fb42fbafb6c336eb123e6a9dcfff69e"
server_key_version: "encryptionkey@0"
I20260807 22:48:37.138453 301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "4df16ca22bc248069ab8864eeab8d81a"
format_stamp: "Formatted at 2026-08-07 22:48:37 on dist-test-slave-rgb1"
server_key: "f432dc09089422c0bddf2d447d9501f9"
server_key_iv: "9fb42fbafb6c336eb123e6a9dcfff69e"
server_key_version: "encryptionkey@0"
I20260807 22:48:37.140098 301 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 22:48:37.141028 316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:37.141291 301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:37.141422 301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "4df16ca22bc248069ab8864eeab8d81a"
format_stamp: "Formatted at 2026-08-07 22:48:37 on dist-test-slave-rgb1"
server_key: "f432dc09089422c0bddf2d447d9501f9"
server_key_iv: "9fb42fbafb6c336eb123e6a9dcfff69e"
server_key_version: "encryptionkey@0"
I20260807 22:48:37.141489 301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:37.149312 301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:37.151703 301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:37.151834 301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:37.152055 301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:37.152329 301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:37.152361 301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:37.152383 301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:37.152396 301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:37.158763 301 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:44679
I20260807 22:48:37.158842 433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:44679 every 8 connection(s)
I20260807 22:48:37.159245 301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:48:37.166355 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 301
I20260807 22:48:37.166515 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:48:37.166756 16778 external_mini_cluster.cc:1428] Setting key de18f62322be08ea97f5076e57bf2bd3
Aug 07 22:48:37 dist-test-slave-rgb1 krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142917, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:37.171376 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.160104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:52349)
0807 22:48:37.160389 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:37.160432 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:37.161095 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:37.161431 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:37.161435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:37.162036 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:37.162306 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:37.162312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.163155 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.163156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:37.164026 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:37.164029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.164489 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.164490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:37.164716 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.165385 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:37.165395 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:37.167324 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:37.169465 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:37.169478 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:37.169480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:37.169798 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:37.170073 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:37.170074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:37.170075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:37.170198 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:37.170538 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:37.170545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:37.170888 (+ 343us) client_negotiation.cc:769] Sending connection context
0807 22:48:37.171015 (+ 127us) client_negotiation.cc:240] Negotiation successful
0807 22:48:37.171184 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":126,"threads_started":1}
I20260807 22:48:37.171329 32607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.160202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:52349 (local address 127.16.98.190:43079)
0807 22:48:37.160288 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:37.160290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:37.160637 (+ 347us) server_negotiation.cc:407] Connection header received
0807 22:48:37.161232 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:37.161233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:37.161246 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:37.161315 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:37.162411 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.163026 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.164151 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.164350 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.164417 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.167548 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:37.167550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:37.167550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:37.167560 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:37.169279 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:37.169876 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:37.169877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:37.169877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:37.169949 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:37.170282 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:37.170283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:37.170283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:37.170453 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:37.170539 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:37.171072 (+ 533us) server_negotiation.cc:299] Negotiation successful
0807 22:48:37.171186 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 22:48:37.171967 434 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43079
I20260807 22:48:37.172188 434 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:37.172412 434 heartbeater.cc:511] Master 127.16.98.190:43079 requested a full tablet report, sending...
I20260807 22:48:37.172971 32566 ts_manager.cc:194] Registered new tserver with Master: 4df16ca22bc248069ab8864eeab8d81a (127.16.98.130:44679)
I20260807 22:48:37.173524 32566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:52349
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:48:37.201794 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43079
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:37.295714 441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:37.295969 441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:37.295998 441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:37.297434 441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:37.297494 441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:37.297526 441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:48:37.299342 441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:37.299664 441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:37.299960 441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:37.306584 447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:37.306592 446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:37.306617 449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:37.306900 441 server_base.cc:1034] running on GCE node
I20260807 22:48:37.307114 441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:37.307395 441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:37.308632 441 hybrid_clock.cc:648] HybridClock initialized: now 1786142917308606 us; error 36 us; skew 500 ppm
Aug 07 22:48:37 dist-test-slave-rgb1 krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142917, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:37.311502 441 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:48:37.311957 441 webserver.cc:460] Webserver started at http://127.16.98.131:36505/ using document root <none> and password file <none>
I20260807 22:48:37.312239 441 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:37.312291 441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:37.312433 441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:48:37.313666 441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "457d288fc4724700944d815621854da1"
format_stamp: "Formatted at 2026-08-07 22:48:37 on dist-test-slave-rgb1"
server_key: "32fdeff36089f73873ac112bddb23976"
server_key_iv: "0bf197a3d1099eee7d1f1c9502bcafc8"
server_key_version: "encryptionkey@0"
I20260807 22:48:37.314086 441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "457d288fc4724700944d815621854da1"
format_stamp: "Formatted at 2026-08-07 22:48:37 on dist-test-slave-rgb1"
server_key: "32fdeff36089f73873ac112bddb23976"
server_key_iv: "0bf197a3d1099eee7d1f1c9502bcafc8"
server_key_version: "encryptionkey@0"
I20260807 22:48:37.315615 441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:48:37.316494 456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:37.316735 441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:37.316851 441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "457d288fc4724700944d815621854da1"
format_stamp: "Formatted at 2026-08-07 22:48:37 on dist-test-slave-rgb1"
server_key: "32fdeff36089f73873ac112bddb23976"
server_key_iv: "0bf197a3d1099eee7d1f1c9502bcafc8"
server_key_version: "encryptionkey@0"
I20260807 22:48:37.316943 441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:37.325886 441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:37.328477 441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:37.328732 441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:37.328987 441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:37.329334 441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:37.329388 441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:37.329432 441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:37.329460 441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:37.335450 441 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:42159
I20260807 22:48:37.335551 572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:42159 every 8 connection(s)
I20260807 22:48:37.335894 441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:48:37.338521 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 441
I20260807 22:48:37.338620 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:48:37.338822 16778 external_mini_cluster.cc:1428] Setting key 18d7c5d94aa3dd1259863b01f798135c
Aug 07 22:48:37 dist-test-slave-rgb1 krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142917, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:37.347549 32607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.336739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:55593 (local address 127.16.98.190:43079)
0807 22:48:37.336816 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:37.336819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:37.337161 (+ 342us) server_negotiation.cc:407] Connection header received
0807 22:48:37.337652 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:37.337654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:37.337665 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:37.337725 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:37.338938 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.339775 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.341138 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.341331 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.341399 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.344236 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:37.344237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:37.344238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:37.344244 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:37.345858 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:37.346491 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:37.346492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:37.346493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:37.346557 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:37.346920 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:37.346921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:37.346921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:37.347049 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:37.347111 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:37.347395 (+ 284us) server_negotiation.cc:299] Negotiation successful
0807 22:48:37.347471 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 22:48:37.347769 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.336676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:55593)
0807 22:48:37.336964 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:37.337017 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:37.337536 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:37.337776 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:37.337780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:37.338475 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:37.338793 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:37.338801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.339922 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.339924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:37.341049 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:37.341052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.341426 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.341427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:37.341601 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.342328 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:37.342337 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:37.343963 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:37.346107 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:37.346115 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:37.346117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:37.346385 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:37.346669 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:37.346670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:37.346670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:37.346812 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:37.347123 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:37.347130 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:37.347266 (+ 136us) client_negotiation.cc:769] Sending connection context
0807 22:48:37.347364 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 22:48:37.347511 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260807 22:48:37.348381 573 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43079
I20260807 22:48:37.348500 573 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:37.348731 573 heartbeater.cc:511] Master 127.16.98.190:43079 requested a full tablet report, sending...
I20260807 22:48:37.349224 32566 ts_manager.cc:194] Registered new tserver with Master: 457d288fc4724700944d815621854da1 (127.16.98.131:42159)
I20260807 22:48:37.349670 32566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:55593
I20260807 22:48:37.350795 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:48:37.359017 32607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.352244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.16.98.190:43079)
0807 22:48:37.352296 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:37.352298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:37.352323 (+ 25us) server_negotiation.cc:407] Connection header received
0807 22:48:37.352401 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:37.352402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:37.352412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:37.352467 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:37.353123 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.353606 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.354403 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.354608 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.354688 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.356070 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:37.356072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:37.356072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:37.356079 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:37.357651 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:37.358102 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:37.358103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:37.358104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:37.358164 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:37.358379 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:37.358380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:37.358380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:37.358601 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:37.358714 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:37.358822 (+ 108us) server_negotiation.cc:299] Negotiation successful
0807 22:48:37.358907 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 22:48:37.361035 32566 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:48:37.362658 32566 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:48:37.370708 587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.365891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37094 (local address 127.16.98.130:44679)
0807 22:48:37.366266 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:37.366271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:37.366282 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:48:37.366330 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:37.366332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:37.366350 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:37.366468 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:37.366631 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.367660 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.369109 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.369921 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.370195 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.370281 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:37.370399 (+ 118us) server_negotiation.cc:299] Negotiation successful
0807 22:48:37.370532 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":88,"threads_started":1}
I20260807 22:48:37.370687 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.365756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:44679 (local address 127.0.0.1:37094)
0807 22:48:37.365970 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:37.366020 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:37.366098 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:37.366478 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:37.366482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:37.366486 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:37.366565 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:37.366567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.367770 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.367771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:37.369005 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:37.369008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.370193 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.370195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:37.370284 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.370299 (+ 15us) client_negotiation.cc:769] Sending connection context
0807 22:48:37.370382 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 22:48:37.370512 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":120,"threads_started":1}
I20260807 22:48:37.371222 589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.366137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.16.98.131:42159)
0807 22:48:37.366661 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:37.366666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:37.366677 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:48:37.366714 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:37.366716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:37.366730 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:37.366815 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:37.367129 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.368438 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.369596 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.370420 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.370504 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.370559 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:37.371078 (+ 519us) server_negotiation.cc:299] Negotiation successful
0807 22:48:37.371137 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":89,"threads_started":1}
I20260807 22:48:37.371408 585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.365761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:42159 (local address 127.0.0.1:38302)
0807 22:48:37.366300 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:37.366311 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:37.366368 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:37.366881 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:37.366883 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:37.366885 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:37.367014 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:37.367017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.368605 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.368605 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:37.369503 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:37.369505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.370598 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.370600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:37.370681 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.370685 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:48:37.370737 (+ 52us) client_negotiation.cc:240] Negotiation successful
0807 22:48:37.370784 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":28,"thread_start_us":85,"threads_started":1}
I20260807 22:48:37.371757 586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.365761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:33245 (local address 127.0.0.1:53288)
0807 22:48:37.366322 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:37.366334 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:37.366396 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:37.366918 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:37.366919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:37.366922 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:37.367089 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:37.367093 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.368612 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.368613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:37.369752 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:37.369756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.371500 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.371501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:37.371560 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.371564 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:48:37.371641 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 22:48:37.371691 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":34,"thread_start_us":104,"threads_started":1}
I20260807 22:48:37.371968 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.366051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53288 (local address 127.16.98.129:33245)
0807 22:48:37.366265 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:37.366271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:37.366656 (+ 385us) server_negotiation.cc:407] Connection header received
0807 22:48:37.366709 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:37.366711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:37.366729 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:37.366815 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:37.367220 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.368416 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.369854 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.371285 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.371358 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.371417 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:37.371863 (+ 446us) server_negotiation.cc:299] Negotiation successful
0807 22:48:37.371902 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
I20260807 22:48:37.372046 365 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac57a91ca5a4e31878d490b5b021105 (DEFAULT_TABLE table=test-table [id=6db8723d21fb43228686737a0487c8a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:48:37.372489 365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac57a91ca5a4e31878d490b5b021105. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:37.372870 505 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac57a91ca5a4e31878d490b5b021105 (DEFAULT_TABLE table=test-table [id=6db8723d21fb43228686737a0487c8a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:48:37.373198 505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac57a91ca5a4e31878d490b5b021105. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:37.373417 32694 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac57a91ca5a4e31878d490b5b021105 (DEFAULT_TABLE table=test-table [id=6db8723d21fb43228686737a0487c8a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:48:37.373816 32694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac57a91ca5a4e31878d490b5b021105. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:37.376477 590 tablet_bootstrap.cc:492] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: Bootstrap starting.
I20260807 22:48:37.376917 592 tablet_bootstrap.cc:492] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1: Bootstrap starting.
I20260807 22:48:37.377349 590 tablet_bootstrap.cc:654] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:37.377573 592 tablet_bootstrap.cc:654] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:37.378023 590 log.cc:824] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:37.378051 592 log.cc:824] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:37.378439 591 tablet_bootstrap.cc:492] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: Bootstrap starting.
I20260807 22:48:37.378999 591 tablet_bootstrap.cc:654] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:37.379016 590 tablet_bootstrap.cc:492] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: No bootstrap required, opened a new log
I20260807 22:48:37.379109 590 ts_tablet_manager.cc:1397] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:48:37.380141 591 log.cc:824] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:37.380971 592 tablet_bootstrap.cc:492] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1: No bootstrap required, opened a new log
I20260807 22:48:37.381071 592 ts_tablet_manager.cc:1397] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260807 22:48:37.381173 591 tablet_bootstrap.cc:492] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: No bootstrap required, opened a new log
I20260807 22:48:37.381263 591 ts_tablet_manager.cc:1397] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:48:37.381512 590 raft_consensus.cc:348] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:37.381737 590 raft_consensus.cc:374] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:37.381775 590 raft_consensus.cc:729] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705e830ad7b74ed6b85e48ec392fb19a, State: Initialized, Role: FOLLOWER
I20260807 22:48:37.381945 590 consensus_queue.cc:260] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:37.382308 590 ts_tablet_manager.cc:1428] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 22:48:37.382362 32762 heartbeater.cc:503] Master 127.16.98.190:43079 was elected leader, sending a full tablet report...
I20260807 22:48:37.383212 591 raft_consensus.cc:348] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:37.383354 591 raft_consensus.cc:374] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:37.383378 591 raft_consensus.cc:729] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df16ca22bc248069ab8864eeab8d81a, State: Initialized, Role: FOLLOWER
I20260807 22:48:37.383508 591 consensus_queue.cc:260] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:37.383483 592 raft_consensus.cc:348] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:37.383670 592 raft_consensus.cc:374] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:37.383723 592 raft_consensus.cc:729] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457d288fc4724700944d815621854da1, State: Initialized, Role: FOLLOWER
I20260807 22:48:37.383839 591 ts_tablet_manager.cc:1428] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:48:37.383900 434 heartbeater.cc:503] Master 127.16.98.190:43079 was elected leader, sending a full tablet report...
I20260807 22:48:37.383870 592 consensus_queue.cc:260] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:37.384244 592 ts_tablet_manager.cc:1428] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260807 22:48:37.384418 573 heartbeater.cc:503] Master 127.16.98.190:43079 was elected leader, sending a full tablet report...
W20260807 22:48:37.409968 436 tablet.cc:2369] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:48:37.474305 597 raft_consensus.cc:482] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:48:37.474543 597 raft_consensus.cc:504] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:37.474997 597 leader_election.cc:288] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 457d288fc4724700944d815621854da1 (127.16.98.131:42159), 705e830ad7b74ed6b85e48ec392fb19a (127.16.98.129:33245)
I20260807 22:48:37.478327 589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.475307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:35199 (local address 127.16.98.131:42159)
0807 22:48:37.475371 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:37.475373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:37.475572 (+ 199us) server_negotiation.cc:407] Connection header received
0807 22:48:37.475657 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:37.475658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:37.475669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:37.475735 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:37.475917 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.476437 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.477424 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.477938 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.478034 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.478042 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:37.478210 (+ 168us) server_negotiation.cc:299] Negotiation successful
0807 22:48:37.478252 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 22:48:37.478340 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.475346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:39797 (local address 127.16.98.129:33245)
0807 22:48:37.475399 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:37.475401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:37.475411 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:48:37.475443 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:37.475444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:37.475462 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:37.475538 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:37.475713 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.476255 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.477364 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.477888 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.477983 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.477994 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:37.478193 (+ 199us) server_negotiation.cc:299] Negotiation successful
0807 22:48:37.478237 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260807 22:48:37.478336 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.475230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:33245 (local address 127.16.98.130:39797)
0807 22:48:37.475297 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:37.475313 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:37.475365 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:37.475546 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:37.475547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:37.475549 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:37.475632 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:37.475634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.476378 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.476379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:37.477251 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:37.477254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.477980 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.477981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:37.478092 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.478096 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:48:37.478184 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 22:48:37.478238 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260807 22:48:37.478685 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.475238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:42159 (local address 127.16.98.130:35199)
0807 22:48:37.475484 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:37.475494 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:37.475583 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:37.475754 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:37.475755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:37.475756 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:37.475841 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:37.475842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.476538 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.476539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:37.477323 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:37.477326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.478034 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.478035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:37.478112 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.478123 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 22:48:37.478190 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 22:48:37.478234 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":15,"thread_start_us":125,"threads_started":1}
I20260807 22:48:37.478780 32715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac57a91ca5a4e31878d490b5b021105" candidate_uuid: "4df16ca22bc248069ab8864eeab8d81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "705e830ad7b74ed6b85e48ec392fb19a" is_pre_election: true
I20260807 22:48:37.479008 32715 raft_consensus.cc:2454] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df16ca22bc248069ab8864eeab8d81a in term 0.
I20260807 22:48:37.479095 526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac57a91ca5a4e31878d490b5b021105" candidate_uuid: "4df16ca22bc248069ab8864eeab8d81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457d288fc4724700944d815621854da1" is_pre_election: true
I20260807 22:48:37.479262 526 raft_consensus.cc:2454] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4df16ca22bc248069ab8864eeab8d81a in term 0.
I20260807 22:48:37.479377 319 leader_election.cc:302] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df16ca22bc248069ab8864eeab8d81a, 705e830ad7b74ed6b85e48ec392fb19a; no voters:
I20260807 22:48:37.479548 597 raft_consensus.cc:2781] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:48:37.479614 597 raft_consensus.cc:482] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:48:37.479632 597 raft_consensus.cc:3037] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:37.480441 597 raft_consensus.cc:504] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:37.480646 597 leader_election.cc:288] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [CANDIDATE]: Term 1 election: Requested vote from peers 457d288fc4724700944d815621854da1 (127.16.98.131:42159), 705e830ad7b74ed6b85e48ec392fb19a (127.16.98.129:33245)
I20260807 22:48:37.480855 526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac57a91ca5a4e31878d490b5b021105" candidate_uuid: "4df16ca22bc248069ab8864eeab8d81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457d288fc4724700944d815621854da1"
I20260807 22:48:37.480963 526 raft_consensus.cc:3037] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:37.481810 526 raft_consensus.cc:2454] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df16ca22bc248069ab8864eeab8d81a in term 1.
I20260807 22:48:37.481969 32715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac57a91ca5a4e31878d490b5b021105" candidate_uuid: "4df16ca22bc248069ab8864eeab8d81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "705e830ad7b74ed6b85e48ec392fb19a"
I20260807 22:48:37.482079 320 leader_election.cc:302] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 457d288fc4724700944d815621854da1, 4df16ca22bc248069ab8864eeab8d81a; no voters:
I20260807 22:48:37.482075 32715 raft_consensus.cc:3037] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:37.482172 597 raft_consensus.cc:2781] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:48:37.482352 597 raft_consensus.cc:686] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 1 LEADER]: Becoming Leader. State: Replica: 4df16ca22bc248069ab8864eeab8d81a, State: Running, Role: LEADER
I20260807 22:48:37.482457 597 consensus_queue.cc:237] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:37.483093 32715 raft_consensus.cc:2454] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4df16ca22bc248069ab8864eeab8d81a in term 1.
I20260807 22:48:37.483466 32565 catalog_manager.cc:5495] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4df16ca22bc248069ab8864eeab8d81a (127.16.98.130). New cstate: current_term: 1 leader_uuid: "4df16ca22bc248069ab8864eeab8d81a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } health_report { overall_health: UNKNOWN } } }
W20260807 22:48:37.489713 32764 tablet.cc:2369] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:48:37.519106 587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.515967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.16.98.130:44679)
0807 22:48:37.516092 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:37.516094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:37.516104 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:48:37.516148 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:37.516149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:37.516171 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:37.516244 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:48:37.516473 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.517040 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.517924 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:37.518167 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:37.518230 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:37.518438 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:48:37.518561 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:48:37.518636 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:37.518864 (+ 228us) server_negotiation.cc:299] Negotiation successful
0807 22:48:37.518901 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 22:48:37.521561 32715 raft_consensus.cc:1261] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a [term 1 FOLLOWER]: Refusing update from remote peer 4df16ca22bc248069ab8864eeab8d81a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:48:37.521561 526 raft_consensus.cc:1261] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1 [term 1 FOLLOWER]: Refusing update from remote peer 4df16ca22bc248069ab8864eeab8d81a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:48:37.522060 597 consensus_queue.cc:1035] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [LEADER]: Connected to new peer: Peer: permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:48:37.522380 601 consensus_queue.cc:1035] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [LEADER]: Connected to new peer: Peer: permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:48:37.524919 604 mvcc.cc:205] Tried to move back new op lower bound from 7316041390164619264 to 7316041390008963072. Current Snapshot: MvccSnapshot[applied={T|T < 7316041390164619264}]
I20260807 22:48:37.525190 605 mvcc.cc:205] Tried to move back new op lower bound from 7316041390164619264 to 7316041390008963072. Current Snapshot: MvccSnapshot[applied={T|T < 7316041390164619264}]
I20260807 22:48:37.526286 606 mvcc.cc:205] Tried to move back new op lower bound from 7316041390164619264 to 7316041390008963072. Current Snapshot: MvccSnapshot[applied={T|T < 7316041390164619264}]
I20260807 22:48:37.527575 32566 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:48:37.527685 32566 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:48:37.528786 32566 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Sending DeleteTablet for 3 replicas of tablet 6ac57a91ca5a4e31878d490b5b021105
I20260807 22:48:37.529175 365 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ac57a91ca5a4e31878d490b5b021105 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:37 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:37094
I20260807 22:48:37.529206 505 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ac57a91ca5a4e31878d490b5b021105 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:37 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:38302
I20260807 22:48:37.529515 615 tablet_replica.cc:330] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: stopping tablet replica
I20260807 22:48:37.529492 32694 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ac57a91ca5a4e31878d490b5b021105 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:37 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:53288
I20260807 22:48:37.529706 615 raft_consensus.cc:2229] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:48:37.529775 617 tablet_replica.cc:330] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: stopping tablet replica
I20260807 22:48:37.529780 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 32631
I20260807 22:48:37.533774 616 tablet_replica.cc:330] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1: stopping tablet replica
I20260807 22:48:37.533947 616 raft_consensus.cc:2229] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:48:37.534094 616 raft_consensus.cc:2258] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:48:37.534669 616 ts_tablet_manager.cc:1905] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:48:37.536937 616 ts_tablet_manager.cc:1918] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:48:37.537053 616 log.cc:1196] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/6ac57a91ca5a4e31878d490b5b021105
I20260807 22:48:37.537375 616 ts_tablet_manager.cc:1939] T 6ac57a91ca5a4e31878d490b5b021105 P 457d288fc4724700944d815621854da1: Deleting consensus metadata
I20260807 22:48:37.538118 32553 catalog_manager.cc:4841] TS 457d288fc4724700944d815621854da1 (127.16.98.131:42159): tablet 6ac57a91ca5a4e31878d490b5b021105 (table test-table [id=6db8723d21fb43228686737a0487c8a7]) successfully deleted
W20260807 22:48:37.538483 32552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:33245 (error 108)
I20260807 22:48:37.539089 586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.538932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:33245 (local address 127.0.0.1:53294)
0807 22:48:37.538999 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:37.539044 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:33245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 22:48:37.539214 32552 catalog_manager.cc:4584] TS 705e830ad7b74ed6b85e48ec392fb19a (127.16.98.129:33245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6ac57a91ca5a4e31878d490b5b021105: Network error: Client connection negotiation failed: client connection to 127.16.98.129:33245: connect: Connection refused (error 111)
I20260807 22:48:37.539510 615 raft_consensus.cc:2258] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:48:37.539776 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 301
I20260807 22:48:37.550302 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 441
I20260807 22:48:37.550501 586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:37.550314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:44679 (local address 127.0.0.1:37114)
0807 22:48:37.550395 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:37.550437 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.130:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 22:48:37.558216 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 32534
I20260807 22:48:57.566109 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43079
--webserver_interface=127.16.98.190
--webserver_port=36629
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:43079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:48:57.658088 619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:57.658373 619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:57.658445 619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:57.659736 619 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 22:48:57.659780 619 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 22:48:57.659794 619 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 22:48:57.659807 619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:48:57.659822 619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:57.659837 619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:57.659852 619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:48:57.659888 619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:48:57.661612 619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:43079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=36629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:57.661844 619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:57.662045 619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:57.671898 627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:57.671885 626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:57.671929 619 server_base.cc:1034] running on GCE node
W20260807 22:48:57.671859 629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:57.672454 619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:57.672835 619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:57.674082 619 hybrid_clock.cc:648] HybridClock initialized: now 1786142937674064 us; error 33 us; skew 500 ppm
Aug 07 22:48:57 dist-test-slave-rgb1 krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142937, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:57.677675 619 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:48:57.678262 619 webserver.cc:460] Webserver started at http://127.16.98.190:36629/ using document root <none> and password file <none>
I20260807 22:48:57.678514 619 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:57.678577 619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:57.680176 619 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:48:57.681133 637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:57.681388 619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 22:48:57.681520 619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "86e3a23ef2354b1d89292ed4831f40c0"
format_stamp: "Formatted at 2026-08-07 22:48:36 on dist-test-slave-rgb1"
server_key: "973af8f4f6a612c2f77c6d796eec8064"
server_key_iv: "ba523c21184d363110ffd5d0b6f22d14"
server_key_version: "encryptionkey@0"
I20260807 22:48:57.681960 619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:57.720849 619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:57.724874 619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:57.725023 619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:57.729818 619 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:43079
I20260807 22:48:57.729893 692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:43079 every 8 connection(s)
I20260807 22:48:57.730371 619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:48:57.732089 693 sys_catalog.cc:263] Verifying existing consensus state
I20260807 22:48:57.732723 693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Bootstrap starting.
I20260807 22:48:57.733650 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 619
I20260807 22:48:57.733952 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:33245
--local_ip_for_outbound_sockets=127.16.98.129
--tserver_master_addrs=127.16.98.190:43079
--webserver_port=33585
--webserver_interface=127.16.98.129
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:48:57.743741 693 log.cc:824] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:57.746153 693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:48:57.746547 693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Bootstrap complete.
I20260807 22:48:57.749362 693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } }
I20260807 22:48:57.749657 693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e3a23ef2354b1d89292ed4831f40c0, State: Initialized, Role: FOLLOWER
I20260807 22:48:57.749842 693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } }
I20260807 22:48:57.749915 693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:48:57.749991 693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:48:57.750044 693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:48:57.751361 693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } }
I20260807 22:48:57.751502 693 leader_election.cc:302] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86e3a23ef2354b1d89292ed4831f40c0; no voters:
I20260807 22:48:57.751757 693 leader_election.cc:288] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 22:48:57.751799 697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 22:48:57.751940 697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 2 LEADER]: Becoming Leader. State: Replica: 86e3a23ef2354b1d89292ed4831f40c0, State: Running, Role: LEADER
I20260807 22:48:57.752081 693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:48:57.752110 697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } }
I20260807 22:48:57.752466 698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "86e3a23ef2354b1d89292ed4831f40c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } } }
I20260807 22:48:57.752467 697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86e3a23ef2354b1d89292ed4831f40c0. Latest consensus state: current_term: 2 leader_uuid: "86e3a23ef2354b1d89292ed4831f40c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } } }
I20260807 22:48:57.752627 698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:57.752646 697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [sys.catalog]: This master's current role is: LEADER
I20260807 22:48:57.752938 703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:48:57.754019 703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:48:57.754221 703 catalog_manager.cc:1259] Loaded cluster ID: 75bcb32455bf41f49b13950f9d8a3352
I20260807 22:48:57.754511 703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:48:57.755419 703 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:48:57.755647 703 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Loaded TSK: 0
I20260807 22:48:57.763792 703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Generated new TSK 1
I20260807 22:48:57.763931 703 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 22:48:57.844607 695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:57.844866 695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:57.844894 695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:57.846264 695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:57.846316 695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:57.846347 695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:48:57.848095 695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:33245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=33585
--tserver_master_addrs=127.16.98.190:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:57.848412 695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:57.848750 695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:57.854578 721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:57.854714 722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:57.854724 695 server_base.cc:1034] running on GCE node
W20260807 22:48:57.854575 724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:57.855191 695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:57.855522 695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:57.856685 695 hybrid_clock.cc:648] HybridClock initialized: now 1786142937856680 us; error 48 us; skew 500 ppm
Aug 07 22:48:57 dist-test-slave-rgb1 krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142937, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:57.859935 695 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:48:57.860471 695 webserver.cc:460] Webserver started at http://127.16.98.129:33585/ using document root <none> and password file <none>
I20260807 22:48:57.860828 695 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:57.860922 695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:57.862491 695 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:48:57.863366 731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:57.863655 695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:57.863786 695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "705e830ad7b74ed6b85e48ec392fb19a"
format_stamp: "Formatted at 2026-08-07 22:48:36 on dist-test-slave-rgb1"
server_key: "b3efa980fa077a4db385ac0a9acc4433"
server_key_iv: "24b1fdc10c2bd1c3b6895d33103976ce"
server_key_version: "encryptionkey@0"
I20260807 22:48:57.864301 695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:57.875104 695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:57.877905 695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:57.878144 695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:57.878414 695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:57.878983 738 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 22:48:57.880612 695 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 22:48:57.880673 695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260807 22:48:57.880702 695 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 22:48:57.881476 695 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 22:48:57.881522 695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260807 22:48:57.881572 738 tablet_bootstrap.cc:492] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: Bootstrap starting.
I20260807 22:48:57.889510 695 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:33245
I20260807 22:48:57.889586 847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:33245 every 8 connection(s)
I20260807 22:48:57.890043 695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:48:57.891536 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 695
I20260807 22:48:57.891857 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:44679
--local_ip_for_outbound_sockets=127.16.98.130
--tserver_master_addrs=127.16.98.190:43079
--webserver_port=43485
--webserver_interface=127.16.98.130
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:48:57.894615 738 log.cc:824] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:57.899423 738 tablet_bootstrap.cc:492] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:48:57.899923 738 tablet_bootstrap.cc:492] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: Bootstrap complete.
I20260807 22:48:57.901976 738 ts_tablet_manager.cc:1397] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.011s
Aug 07 22:48:57 dist-test-slave-rgb1 krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142937, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:57.904322 738 raft_consensus.cc:348] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:57.904812 738 raft_consensus.cc:729] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705e830ad7b74ed6b85e48ec392fb19a, State: Initialized, Role: FOLLOWER
I20260807 22:48:57.905081 738 consensus_queue.cc:260] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:57.905498 738 ts_tablet_manager.cc:1428] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:48:57.907320 851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:57.891096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:47725 (local address 127.16.98.190:43079)
0807 22:48:57.892719 (+ 1623us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:57.892730 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:57.893009 (+ 279us) server_negotiation.cc:407] Connection header received
0807 22:48:57.894026 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:57.894036 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:57.894100 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:57.894215 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:57.895356 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:57.896937 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:57.898686 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:57.899348 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:57.899438 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:57.902735 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:57.902741 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:57.902741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:57.902756 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:57.905247 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:57.905820 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:57.905823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:57.905824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:57.905940 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:57.906221 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:57.906222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:57.906222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:57.906520 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:57.906579 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:57.906955 (+ 376us) server_negotiation.cc:299] Negotiation successful
0807 22:48:57.907106 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1559,"thread_start_us":180,"threads_started":1}
I20260807 22:48:57.907788 850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:57.890995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:47725)
0807 22:48:57.892833 (+ 1838us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:57.892884 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:57.893748 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:57.894470 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:57.894475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:57.894925 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:57.895214 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:57.895222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:57.897104 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:57.897106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:57.898567 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:57.898572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:57.899613 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:57.899614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:57.899782 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:57.900477 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:57.900487 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:57.902578 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:57.905358 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:57.905365 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:57.905368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:57.905716 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:57.906026 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:57.906027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:57.906027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:57.906148 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:57.906631 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:57.906637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:57.906778 (+ 141us) client_negotiation.cc:769] Sending connection context
0807 22:48:57.907424 (+ 646us) client_negotiation.cc:240] Negotiation successful
0807 22:48:57.907610 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1784,"thread_start_us":109,"threads_started":1}
I20260807 22:48:57.908550 848 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43079
I20260807 22:48:57.908691 848 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:57.908886 848 heartbeater.cc:511] Master 127.16.98.190:43079 requested a full tablet report, sending...
I20260807 22:48:57.909673 655 ts_manager.cc:194] Registered new tserver with Master: 705e830ad7b74ed6b85e48ec392fb19a (127.16.98.129:33245)
I20260807 22:48:57.910962 655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:47725
I20260807 22:48:57.912477 848 heartbeater.cc:503] Master 127.16.98.190:43079 was elected leader, sending a full tablet report...
Aug 07 22:48:57 dist-test-slave-rgb1 krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142937, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for kudu/127.16.98.129@KRBTEST.COM
I20260807 22:48:57.922526 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:57.911277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.16.98.129:33245)
0807 22:48:57.911995 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:57.911999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:57.912009 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:48:57.912044 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:57.912046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:57.912061 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:57.912127 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:57.913032 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:57.914194 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:57.915448 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:57.916165 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:57.916250 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:57.919119 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:57.919128 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:57.919129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:57.919137 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:57.921110 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:57.921592 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:57.921595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:57.921596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:57.921664 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:57.921883 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:57.921884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:57.921884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:57.922051 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:57.922137 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:57.922333 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 22:48:57.922421 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":108,"threads_started":1}
I20260807 22:48:57.922541 855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:57.910321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:33245 (local address 127.0.0.1:49814)
0807 22:48:57.911710 (+ 1389us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:57.911752 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:57.911833 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:57.912159 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:57.912164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:57.912520 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:57.912731 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:57.912733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:57.914297 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:57.914298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:57.915339 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:57.915343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:57.916258 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:57.916259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:57.916348 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:57.917008 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:57.917014 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:57.918995 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:57.921203 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:57.921209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:57.921211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:57.921497 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:57.921742 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:57.921743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:57.921743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:57.921815 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:57.922127 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:57.922134 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:57.922193 (+ 59us) client_negotiation.cc:769] Sending connection context
0807 22:48:57.922285 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 22:48:57.922421 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1314,"thread_start_us":117,"threads_started":1}
I20260807 22:48:57.923398 780 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ac57a91ca5a4e31878d490b5b021105 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:37 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:49814
I20260807 22:48:57.924045 857 tablet_replica.cc:330] stopping tablet replica
I20260807 22:48:57.924162 857 raft_consensus.cc:2229] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:48:57.924232 857 raft_consensus.cc:2258] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:48:57.924635 857 ts_tablet_manager.cc:1905] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:48:57.927500 857 ts_tablet_manager.cc:1918] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:48:57.927582 857 log.cc:1196] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal/wals/6ac57a91ca5a4e31878d490b5b021105
I20260807 22:48:57.927877 857 ts_tablet_manager.cc:1939] T 6ac57a91ca5a4e31878d490b5b021105 P 705e830ad7b74ed6b85e48ec392fb19a: Deleting consensus metadata
I20260807 22:48:57.928481 640 catalog_manager.cc:4841] TS 705e830ad7b74ed6b85e48ec392fb19a (127.16.98.129:33245): tablet 6ac57a91ca5a4e31878d490b5b021105 (table test-table [id=6db8723d21fb43228686737a0487c8a7]) successfully deleted
W20260807 22:48:57.987836 852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:57.988101 852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:57.988145 852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:57.989444 852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:57.989521 852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:57.989570 852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:48:57.991117 852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:44679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=43485
--tserver_master_addrs=127.16.98.190:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:57.991413 852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:57.991662 852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:57.997332 864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:57.997273 863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:57.997273 866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:57.997700 852 server_base.cc:1034] running on GCE node
I20260807 22:48:57.997972 852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:57.998296 852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:57.999574 852 hybrid_clock.cc:648] HybridClock initialized: now 1786142937999557 us; error 35 us; skew 500 ppm
Aug 07 22:48:57 dist-test-slave-rgb1 krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142937, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:58.002966 852 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:48:58.003516 852 webserver.cc:460] Webserver started at http://127.16.98.130:43485/ using document root <none> and password file <none>
I20260807 22:48:58.003810 852 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:58.003907 852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:58.005484 852 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:48:58.007119 873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:58.007434 852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260807 22:48:58.007575 852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "4df16ca22bc248069ab8864eeab8d81a"
format_stamp: "Formatted at 2026-08-07 22:48:37 on dist-test-slave-rgb1"
server_key: "f432dc09089422c0bddf2d447d9501f9"
server_key_iv: "9fb42fbafb6c336eb123e6a9dcfff69e"
server_key_version: "encryptionkey@0"
I20260807 22:48:58.008023 852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:58.050690 852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:58.053829 852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:58.054016 852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:58.054292 852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:58.054837 880 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 22:48:58.056429 852 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 22:48:58.056515 852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260807 22:48:58.056593 852 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 22:48:58.057344 852 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 22:48:58.057425 852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260807 22:48:58.057451 880 tablet_bootstrap.cc:492] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: Bootstrap starting.
I20260807 22:48:58.066270 852 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:44679
I20260807 22:48:58.066356 993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:44679 every 8 connection(s)
I20260807 22:48:58.066890 852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:48:58.069972 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 852
I20260807 22:48:58.070284 880 log.cc:824] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:58.070323 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:42159
--local_ip_for_outbound_sockets=127.16.98.131
--tserver_master_addrs=127.16.98.190:43079
--webserver_port=36505
--webserver_interface=127.16.98.131
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:48:58.074028 880 tablet_bootstrap.cc:492] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:48:58.074371 880 tablet_bootstrap.cc:492] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: Bootstrap complete.
Aug 07 22:48:58 dist-test-slave-rgb1 krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142937, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:58.075639 880 ts_tablet_manager.cc:1397] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.013s
I20260807 22:48:58.077440 880 raft_consensus.cc:348] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:58.077872 880 raft_consensus.cc:729] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df16ca22bc248069ab8864eeab8d81a, State: Initialized, Role: FOLLOWER
I20260807 22:48:58.078110 880 consensus_queue.cc:260] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:58.078466 880 ts_tablet_manager.cc:1428] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20260807 22:48:58.080159 851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.067914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:47527 (local address 127.16.98.190:43079)
0807 22:48:58.067971 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:58.067973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:58.068826 (+ 853us) server_negotiation.cc:407] Connection header received
0807 22:48:58.069469 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:58.069470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:58.069484 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:58.069583 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:58.070590 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.071642 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.073208 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.073465 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.073575 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.076330 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:58.076352 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:58.076353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:58.076365 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:58.078482 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:58.079035 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:58.079037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:58.079038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:58.079106 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:58.079406 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:58.079407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:58.079408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:58.079549 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:58.079630 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:58.079951 (+ 321us) server_negotiation.cc:299] Negotiation successful
0807 22:48:58.080052 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 22:48:58.080343 996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.067835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:47527)
0807 22:48:58.068682 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:58.068731 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:58.069354 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:58.069693 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:58.069697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:58.070145 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:58.070456 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:58.070464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.071820 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.071822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:58.073097 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:58.073100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.073611 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.073613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:58.073766 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.074471 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:58.074482 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:58.076224 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:58.078616 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:58.078623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:58.078625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:58.078940 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:58.079188 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:58.079189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:58.079190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:58.079326 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:58.079634 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:58.079642 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:58.079795 (+ 153us) client_negotiation.cc:769] Sending connection context
0807 22:48:58.079913 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 22:48:58.080109 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":106,"threads_started":1}
I20260807 22:48:58.080935 994 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43079
I20260807 22:48:58.081044 994 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:58.081256 994 heartbeater.cc:511] Master 127.16.98.190:43079 requested a full tablet report, sending...
I20260807 22:48:58.081872 655 ts_manager.cc:194] Registered new tserver with Master: 4df16ca22bc248069ab8864eeab8d81a (127.16.98.130:44679)
I20260807 22:48:58.082577 655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:47527
I20260807 22:48:58.083660 994 heartbeater.cc:503] Master 127.16.98.190:43079 was elected leader, sending a full tablet report...
Aug 07 22:48:58 dist-test-slave-rgb1 krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142937, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for kudu/127.16.98.130@KRBTEST.COM
I20260807 22:48:58.090378 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.082291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49416 (local address 127.16.98.130:44679)
0807 22:48:58.082458 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:58.082463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:58.082472 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:48:58.082521 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:58.082523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:58.082538 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:58.082626 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:58.082960 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.083690 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.084667 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.085007 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.085097 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.087449 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:58.087458 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:58.087458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:58.087469 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:58.089151 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:58.089537 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:58.089540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:58.089540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:58.089594 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:58.089802 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:58.089803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:58.089803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:58.089959 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:58.090056 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:58.090197 (+ 141us) server_negotiation.cc:299] Negotiation successful
0807 22:48:58.090292 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260807 22:48:58.090377 855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.082222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:44679 (local address 127.0.0.1:49416)
0807 22:48:58.082283 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:58.082294 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:58.082340 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:58.082626 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:58.082627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:58.082818 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:58.082870 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:58.082872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.083828 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.083830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:58.084536 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:58.084539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.085111 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.085113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:58.085186 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.085702 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:58.085708 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:58.087348 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:58.089254 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:58.089255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:58.089256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:58.089431 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:58.089671 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:58.089672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:58.089672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:58.089737 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:58.090039 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:58.090040 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:58.090087 (+ 47us) client_negotiation.cc:769] Sending connection context
0807 22:48:58.090159 (+ 72us) client_negotiation.cc:240] Negotiation successful
0807 22:48:58.090272 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260807 22:48:58.090776 921 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ac57a91ca5a4e31878d490b5b021105 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:37 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:49416
I20260807 22:48:58.091317 1001 tablet_replica.cc:330] stopping tablet replica
I20260807 22:48:58.091461 1001 raft_consensus.cc:2229] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:48:58.091540 1001 raft_consensus.cc:2258] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:48:58.091898 1001 ts_tablet_manager.cc:1905] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:48:58.094708 1001 ts_tablet_manager.cc:1918] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:48:58.094797 1001 log.cc:1196] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/6ac57a91ca5a4e31878d490b5b021105
I20260807 22:48:58.095052 1001 ts_tablet_manager.cc:1939] T 6ac57a91ca5a4e31878d490b5b021105 P 4df16ca22bc248069ab8864eeab8d81a: Deleting consensus metadata
I20260807 22:48:58.095767 638 catalog_manager.cc:4841] TS 4df16ca22bc248069ab8864eeab8d81a (127.16.98.130:44679): tablet 6ac57a91ca5a4e31878d490b5b021105 (table test-table [id=6db8723d21fb43228686737a0487c8a7]) successfully deleted
W20260807 22:48:58.163703 998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:48:58.164009 998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:48:58.164053 998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:48:58.165390 998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:48:58.165436 998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:48:58.165463 998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:48:58.167361 998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:42159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=36505
--tserver_master_addrs=127.16.98.190:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:48:58.167696 998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:48:58.168032 998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:48:58.174263 1007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:58.174325 1006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:48:58.174288 1009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:48:58.174590 998 server_base.cc:1034] running on GCE node
I20260807 22:48:58.174850 998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:48:58.175189 998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:48:58.176384 998 hybrid_clock.cc:648] HybridClock initialized: now 1786142938176368 us; error 29 us; skew 500 ppm
Aug 07 22:48:58 dist-test-slave-rgb1 krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142938, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:48:58.179517 998 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:48:58.179996 998 webserver.cc:460] Webserver started at http://127.16.98.131:36505/ using document root <none> and password file <none>
I20260807 22:48:58.180230 998 fs_manager.cc:359] Metadata directory not provided
I20260807 22:48:58.180315 998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:48:58.181986 998 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:58.182783 1016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:58.183084 998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:58.183207 998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "457d288fc4724700944d815621854da1"
format_stamp: "Formatted at 2026-08-07 22:48:37 on dist-test-slave-rgb1"
server_key: "32fdeff36089f73873ac112bddb23976"
server_key_iv: "0bf197a3d1099eee7d1f1c9502bcafc8"
server_key_version: "encryptionkey@0"
I20260807 22:48:58.183619 998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:48:58.197535 998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:48:58.200390 998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:48:58.200637 998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:48:58.200942 998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:48:58.201310 998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:48:58.201373 998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:58.201409 998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:48:58.201447 998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:58.208410 998 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:42159
I20260807 22:48:58.208448 1133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:42159 every 8 connection(s)
I20260807 22:48:58.208930 998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
Aug 07 22:48:58 dist-test-slave-rgb1 krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142938, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:48:58.218663 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 998
I20260807 22:48:58.220212 851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.209826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:50115 (local address 127.16.98.190:43079)
0807 22:48:58.209919 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:58.209922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:58.210167 (+ 245us) server_negotiation.cc:407] Connection header received
0807 22:48:58.210755 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:58.210756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:58.210768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:58.210831 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:58.212029 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.212775 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.213939 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.214172 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.214241 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.216813 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:58.216815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:58.216815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:58.216823 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:58.218330 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:58.218927 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:58.218928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:58.218929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:58.219018 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:58.219383 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:58.219384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:58.219385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:58.219575 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:58.219680 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:58.219968 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 22:48:58.220081 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 22:48:58.220234 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.209817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:50115)
0807 22:48:58.210021 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:58.210062 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:58.210639 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:58.210911 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:58.210915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:58.211700 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:48:58.211919 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:58.211925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.212897 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.212898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:58.213832 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:58.213834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.214273 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.214274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:58.214396 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.215008 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:48:58.215018 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:48:58.216678 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:48:58.218487 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:58.218494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:58.218496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:58.218798 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:58.219136 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:48:58.219138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:48:58.219138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:48:58.219294 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:48:58.219665 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:48:58.219672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:48:58.219815 (+ 143us) client_negotiation.cc:769] Sending connection context
0807 22:48:58.219911 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 22:48:58.220057 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":109,"threads_started":1}
I20260807 22:48:58.220857 1134 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43079
I20260807 22:48:58.220990 1134 heartbeater.cc:464] Registering TS with master...
I20260807 22:48:58.221206 1134 heartbeater.cc:511] Master 127.16.98.190:43079 requested a full tablet report, sending...
I20260807 22:48:58.221736 655 ts_manager.cc:194] Registered new tserver with Master: 457d288fc4724700944d815621854da1 (127.16.98.131:42159)
I20260807 22:48:58.222278 655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:50115
I20260807 22:48:58.227250 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.219142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.16.98.190:43079)
0807 22:48:58.219288 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:58.219291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:58.219448 (+ 157us) server_negotiation.cc:407] Connection header received
0807 22:48:58.219513 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:58.219514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:58.219526 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:58.219597 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:58.220644 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.221531 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.222799 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.222991 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.223059 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.224433 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:58.224435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:58.224435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:58.224441 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:58.226047 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:58.226431 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:58.226432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:58.226433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:58.226496 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:58.226701 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:58.226702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:58.226702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:58.226823 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:58.226991 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:58.227058 (+ 67us) server_negotiation.cc:299] Negotiation successful
0807 22:48:58.227142 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260807 22:48:58.228134 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:48:58.231056 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.228591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.16.98.190:43079)
0807 22:48:58.228634 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:58.228636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:58.228737 (+ 101us) server_negotiation.cc:407] Connection header received
0807 22:48:58.228787 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:58.228788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:58.228826 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:58.228893 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:48:58.229049 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.229516 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.230278 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.230473 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.230530 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.230736 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:48:58.230782 (+ 46us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0807 22:48:58.230949 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59490: token expired
Metrics: {"server-negotiator.queue_time_us":18}
W20260807 22:48:58.231112 1137 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59490: token expired
W20260807 22:48:58.231299 577 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.98.190:43079)
I20260807 22:48:58.237632 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.231606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.16.98.190:43079)
0807 22:48:58.231651 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:58.231653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:58.231661 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:48:58.231687 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:58.231687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:58.231704 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:58.231759 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:48:58.232223 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.232797 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.233507 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.233695 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.233754 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.235024 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:48:58.235025 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:48:58.235026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:48:58.235033 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:48:58.236444 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:58.236891 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:58.236892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:58.236893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:58.237015 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:48:58.237211 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:48:58.237212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:48:58.237212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:48:58.237318 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:48:58.237414 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:58.237494 (+ 80us) server_negotiation.cc:299] Negotiation successful
0807 22:48:58.237563 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260807 22:48:58.239081 655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:48:58.240845 655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:48:58.243191 780 tablet_service.cc:1467] Processing CreateTablet for tablet bfc44cd0757b499d92f1b154fc8fae1b (DEFAULT_TABLE table=test-table [id=62ab7420716c4a2a97e66dfe2ea24a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:48:58.243250 921 tablet_service.cc:1467] Processing CreateTablet for tablet bfc44cd0757b499d92f1b154fc8fae1b (DEFAULT_TABLE table=test-table [id=62ab7420716c4a2a97e66dfe2ea24a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:48:58.243410 780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc44cd0757b499d92f1b154fc8fae1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:58.243451 921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc44cd0757b499d92f1b154fc8fae1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:58.245119 738 tablet_bootstrap.cc:492] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a: Bootstrap starting.
I20260807 22:48:58.245142 880 tablet_bootstrap.cc:492] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a: Bootstrap starting.
I20260807 22:48:58.245451 880 tablet_bootstrap.cc:654] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:58.245451 738 tablet_bootstrap.cc:654] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:58.246119 880 tablet_bootstrap.cc:492] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a: No bootstrap required, opened a new log
I20260807 22:48:58.246186 880 ts_tablet_manager.cc:1397] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260807 22:48:58.246448 738 tablet_bootstrap.cc:492] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a: No bootstrap required, opened a new log
I20260807 22:48:58.246508 738 ts_tablet_manager.cc:1397] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260807 22:48:58.246547 880 raft_consensus.cc:348] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:58.246636 880 raft_consensus.cc:374] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:58.246692 880 raft_consensus.cc:729] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df16ca22bc248069ab8864eeab8d81a, State: Initialized, Role: FOLLOWER
I20260807 22:48:58.246660 738 raft_consensus.cc:348] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:58.246728 738 raft_consensus.cc:374] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:58.246780 738 raft_consensus.cc:729] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705e830ad7b74ed6b85e48ec392fb19a, State: Initialized, Role: FOLLOWER
I20260807 22:48:58.246753 880 consensus_queue.cc:260] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:58.246824 738 consensus_queue.cc:260] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:58.246850 880 ts_tablet_manager.cc:1428] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260807 22:48:58.246914 738 ts_tablet_manager.cc:1428] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260807 22:48:58.247588 855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.242938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:42159 (local address 127.0.0.1:54438)
0807 22:48:58.242974 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:58.242987 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:58.243027 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:58.243535 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:58.243537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:58.243539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:58.243590 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:58.243592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.244884 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.244886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:58.246218 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:58.246221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.247285 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.247286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:58.247352 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.247376 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 22:48:58.247460 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 22:48:58.247512 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260807 22:48:58.247608 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.243096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.16.98.131:42159)
0807 22:48:58.243334 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:58.243339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:58.243350 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:48:58.243392 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:58.243394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:58.243411 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:58.243505 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:58.243669 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.244786 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.246351 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.247204 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.247289 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.247358 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:58.247479 (+ 121us) server_negotiation.cc:299] Negotiation successful
0807 22:48:58.247522 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":131,"threads_started":1}
I20260807 22:48:58.248394 1065 tablet_service.cc:1467] Processing CreateTablet for tablet bfc44cd0757b499d92f1b154fc8fae1b (DEFAULT_TABLE table=test-table [id=62ab7420716c4a2a97e66dfe2ea24a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:48:58.248791 1065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc44cd0757b499d92f1b154fc8fae1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:48:58.251969 1143 tablet_bootstrap.cc:492] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1: Bootstrap starting.
I20260807 22:48:58.252867 1143 tablet_bootstrap.cc:654] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1: Neither blocks nor log segments found. Creating new log.
I20260807 22:48:58.253720 1143 log.cc:824] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1: Log is configured to *not* fsync() on all Append() calls
I20260807 22:48:58.254730 1143 tablet_bootstrap.cc:492] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1: No bootstrap required, opened a new log
I20260807 22:48:58.254820 1143 ts_tablet_manager.cc:1397] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:48:58.256282 1143 raft_consensus.cc:348] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:58.256458 1143 raft_consensus.cc:374] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:48:58.256507 1143 raft_consensus.cc:729] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457d288fc4724700944d815621854da1, State: Initialized, Role: FOLLOWER
I20260807 22:48:58.256708 1143 consensus_queue.cc:260] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:58.257058 1143 ts_tablet_manager.cc:1428] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 22:48:58.257102 1134 heartbeater.cc:503] Master 127.16.98.190:43079 was elected leader, sending a full tablet report...
W20260807 22:48:58.317728 995 tablet.cc:2369] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:48:58.368501 1145 raft_consensus.cc:482] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:48:58.368716 1145 raft_consensus.cc:504] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:58.369591 1145 leader_election.cc:288] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4df16ca22bc248069ab8864eeab8d81a (127.16.98.130:44679), 705e830ad7b74ed6b85e48ec392fb19a (127.16.98.129:33245)
I20260807 22:48:58.372455 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.369370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:44679 (local address 127.16.98.131:37835)
0807 22:48:58.369433 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:58.369445 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:58.369490 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:58.369783 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:58.369784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:58.369786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:58.369864 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:58.369866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.370754 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.370755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:58.371573 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:58.371575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.372189 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.372190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:58.372270 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.372274 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:48:58.372337 (+ 63us) client_negotiation.cc:240] Negotiation successful
0807 22:48:58.372381 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260807 22:48:58.372457 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.369463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:37835 (local address 127.16.98.130:44679)
0807 22:48:58.369620 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:58.369623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:58.369632 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:48:58.369664 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:58.369665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:58.369694 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:58.369758 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:58.369935 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.370665 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.371677 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.372105 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.372193 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.372276 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:58.372344 (+ 68us) server_negotiation.cc:299] Negotiation successful
0807 22:48:58.372385 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260807 22:48:58.372839 945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc44cd0757b499d92f1b154fc8fae1b" candidate_uuid: "457d288fc4724700944d815621854da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df16ca22bc248069ab8864eeab8d81a" is_pre_election: true
I20260807 22:48:58.373025 945 raft_consensus.cc:2454] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 457d288fc4724700944d815621854da1 in term 0.
I20260807 22:48:58.373348 1017 leader_election.cc:302] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 457d288fc4724700944d815621854da1, 4df16ca22bc248069ab8864eeab8d81a; no voters:
I20260807 22:48:58.373495 1145 raft_consensus.cc:2781] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:48:58.373538 1145 raft_consensus.cc:482] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:48:58.373603 1145 raft_consensus.cc:3037] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:58.373610 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.369923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:37641 (local address 127.16.98.129:33245)
0807 22:48:58.369984 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:58.369986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:58.370244 (+ 258us) server_negotiation.cc:407] Connection header received
0807 22:48:58.370306 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:58.370307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:58.370336 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:58.370408 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:48:58.371076 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.371611 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.372583 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.373199 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.373291 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.373366 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:58.373478 (+ 112us) server_negotiation.cc:299] Negotiation successful
0807 22:48:58.373538 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260807 22:48:58.373852 1146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.369934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:33245 (local address 127.16.98.131:37641)
0807 22:48:58.370150 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:58.370163 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:48:58.370240 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:48:58.370529 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:48:58.370532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:48:58.370537 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:48:58.370664 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:58.370667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.371712 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.371713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:58.372467 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:48:58.372469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.373304 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.373305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:48:58.373380 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.373384 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:48:58.373740 (+ 356us) client_negotiation.cc:240] Negotiation successful
0807 22:48:58.373795 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":132,"threads_started":1}
I20260807 22:48:58.374197 801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc44cd0757b499d92f1b154fc8fae1b" candidate_uuid: "457d288fc4724700944d815621854da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "705e830ad7b74ed6b85e48ec392fb19a" is_pre_election: true
I20260807 22:48:58.374390 801 raft_consensus.cc:2454] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 457d288fc4724700944d815621854da1 in term 0.
I20260807 22:48:58.374651 1145 raft_consensus.cc:504] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:58.374820 1145 leader_election.cc:288] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [CANDIDATE]: Term 1 election: Requested vote from peers 4df16ca22bc248069ab8864eeab8d81a (127.16.98.130:44679), 705e830ad7b74ed6b85e48ec392fb19a (127.16.98.129:33245)
I20260807 22:48:58.375072 945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc44cd0757b499d92f1b154fc8fae1b" candidate_uuid: "457d288fc4724700944d815621854da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df16ca22bc248069ab8864eeab8d81a"
I20260807 22:48:58.375164 945 raft_consensus.cc:3037] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:58.376535 945 raft_consensus.cc:2454] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 457d288fc4724700944d815621854da1 in term 1.
I20260807 22:48:58.376684 801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc44cd0757b499d92f1b154fc8fae1b" candidate_uuid: "457d288fc4724700944d815621854da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "705e830ad7b74ed6b85e48ec392fb19a"
I20260807 22:48:58.376919 1017 leader_election.cc:302] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 457d288fc4724700944d815621854da1, 4df16ca22bc248069ab8864eeab8d81a; no voters:
I20260807 22:48:58.377032 1145 raft_consensus.cc:2781] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:48:58.376968 801 raft_consensus.cc:3037] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:48:58.377259 1145 raft_consensus.cc:686] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [term 1 LEADER]: Becoming Leader. State: Replica: 457d288fc4724700944d815621854da1, State: Running, Role: LEADER
I20260807 22:48:58.377379 1145 consensus_queue.cc:237] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } } peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } }
I20260807 22:48:58.378274 801 raft_consensus.cc:2454] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 457d288fc4724700944d815621854da1 in term 1.
I20260807 22:48:58.378549 655 catalog_manager.cc:5495] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 457d288fc4724700944d815621854da1 (127.16.98.131). New cstate: current_term: 1 leader_uuid: "457d288fc4724700944d815621854da1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457d288fc4724700944d815621854da1" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 } health_report { overall_health: UNKNOWN } } }
W20260807 22:48:58.391251 849 tablet.cc:2369] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:48:58.398438 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.395808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.16.98.131:42159)
0807 22:48:58.395917 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0807 22:48:58.395919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:48:58.395926 (+ 7us) server_negotiation.cc:407] Connection header received
0807 22:48:58.395975 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:48:58.395976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:48:58.395994 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:48:58.396132 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:48:58.396272 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.396814 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.397593 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:48:58.397803 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:48:58.397884 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:48:58.398059 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:48:58.398176 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:48:58.398277 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 22:48:58.398302 (+ 25us) server_negotiation.cc:299] Negotiation successful
0807 22:48:58.398323 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 22:48:58.400715 945 raft_consensus.cc:1261] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a [term 1 FOLLOWER]: Refusing update from remote peer 457d288fc4724700944d815621854da1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:48:58.400758 801 raft_consensus.cc:1261] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a [term 1 FOLLOWER]: Refusing update from remote peer 457d288fc4724700944d815621854da1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:48:58.401173 1145 consensus_queue.cc:1035] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4df16ca22bc248069ab8864eeab8d81a" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 44679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:48:58.401355 1147 consensus_queue.cc:1035] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "705e830ad7b74ed6b85e48ec392fb19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 33245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:48:58.404229 1152 mvcc.cc:205] Tried to move back new op lower bound from 7316041475685892096 to 7316041475595005952. Current Snapshot: MvccSnapshot[applied={T|T < 7316041475685892096 or (T in {7316041475685892096})}]
I20260807 22:48:58.406764 654 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:48:58.406864 654 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:48:58.407748 654 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Sending DeleteTablet for 3 replicas of tablet bfc44cd0757b499d92f1b154fc8fae1b
I20260807 22:48:58.408149 1065 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc44cd0757b499d92f1b154fc8fae1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:58 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:54438
I20260807 22:48:58.408209 921 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc44cd0757b499d92f1b154fc8fae1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:58 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:49416
I20260807 22:48:58.408181 780 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc44cd0757b499d92f1b154fc8fae1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:48:58 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:49814
I20260807 22:48:58.408321 1001 tablet_replica.cc:330] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a: stopping tablet replica
I20260807 22:48:58.408386 1001 raft_consensus.cc:2229] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:48:58.408439 1160 tablet_replica.cc:330] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1: stopping tablet replica
I20260807 22:48:58.408466 1001 raft_consensus.cc:2258] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:48:58.408548 1160 raft_consensus.cc:2229] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:48:58.408596 857 tablet_replica.cc:330] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a: stopping tablet replica
I20260807 22:48:58.408655 857 raft_consensus.cc:2229] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:48:58.408700 857 raft_consensus.cc:2258] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:48:58.408731 1160 raft_consensus.cc:2258] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:48:58.409108 1001 ts_tablet_manager.cc:1905] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:48:58.409123 857 ts_tablet_manager.cc:1905] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:48:58.409338 1160 ts_tablet_manager.cc:1905] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:48:58.411077 857 ts_tablet_manager.cc:1918] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:48:58.411144 857 log.cc:1196] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal/wals/bfc44cd0757b499d92f1b154fc8fae1b
I20260807 22:48:58.411199 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 695
I20260807 22:48:58.411430 857 ts_tablet_manager.cc:1939] T bfc44cd0757b499d92f1b154fc8fae1b P 705e830ad7b74ed6b85e48ec392fb19a: Deleting consensus metadata
I20260807 22:48:58.411653 1001 ts_tablet_manager.cc:1918] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:48:58.411707 1001 log.cc:1196] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/bfc44cd0757b499d92f1b154fc8fae1b
I20260807 22:48:58.411957 1001 ts_tablet_manager.cc:1939] T bfc44cd0757b499d92f1b154fc8fae1b P 4df16ca22bc248069ab8864eeab8d81a: Deleting consensus metadata
I20260807 22:48:58.412191 1160 ts_tablet_manager.cc:1918] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:48:58.412331 1160 log.cc:1196] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/bfc44cd0757b499d92f1b154fc8fae1b
I20260807 22:48:58.412355 638 catalog_manager.cc:4841] TS 4df16ca22bc248069ab8864eeab8d81a (127.16.98.130:44679): tablet bfc44cd0757b499d92f1b154fc8fae1b (table test-table [id=62ab7420716c4a2a97e66dfe2ea24a95]) successfully deleted
I20260807 22:48:58.412783 1160 ts_tablet_manager.cc:1939] T bfc44cd0757b499d92f1b154fc8fae1b P 457d288fc4724700944d815621854da1: Deleting consensus metadata
I20260807 22:48:58.413295 641 catalog_manager.cc:4841] TS 457d288fc4724700944d815621854da1 (127.16.98.131:42159): tablet bfc44cd0757b499d92f1b154fc8fae1b (table test-table [id=62ab7420716c4a2a97e66dfe2ea24a95]) successfully deleted
W20260807 22:48:58.419592 640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:33245 (error 108)
I20260807 22:48:58.420099 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 852
I20260807 22:48:58.420307 855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:48:58.420159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:33245 (local address 127.0.0.1:49824)
0807 22:48:58.420224 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0807 22:48:58.420268 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:33245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 22:48:58.420435 640 catalog_manager.cc:4584] TS 705e830ad7b74ed6b85e48ec392fb19a (127.16.98.129:33245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bfc44cd0757b499d92f1b154fc8fae1b: Network error: Client connection negotiation failed: client connection to 127.16.98.129:33245: connect: Connection refused (error 111)
I20260807 22:48:58.429826 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 998
I20260807 22:48:58.435906 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 619
I20260807 22:49:18.443742 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43079
--webserver_interface=127.16.98.190
--webserver_port=36629
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:43079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:18.536233 1162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:18.536479 1162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:18.536509 1162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:18.538090 1162 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 22:49:18.538146 1162 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 22:49:18.538160 1162 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 22:49:18.538182 1162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:18.538200 1162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:18.538216 1162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:18.538230 1162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:18.538244 1162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:18.539953 1162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:43079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=36629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:18.540199 1162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:18.540397 1162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 22:49:18.546466 1162 server_base.cc:1034] running on GCE node
W20260807 22:49:18.546432 1167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:18.546482 1168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:18.546427 1170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:18.546944 1162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:18.547304 1162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:18.548503 1162 hybrid_clock.cc:648] HybridClock initialized: now 1786142958548492 us; error 42 us; skew 500 ppm
Aug 07 22:49:18 dist-test-slave-rgb1 krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142958, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:18.551343 1162 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:49:18.551887 1162 webserver.cc:460] Webserver started at http://127.16.98.190:36629/ using document root <none> and password file <none>
I20260807 22:49:18.552162 1162 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:18.552235 1162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:18.553736 1162 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:18.554612 1177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:18.554876 1162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:18.554991 1162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "86e3a23ef2354b1d89292ed4831f40c0"
format_stamp: "Formatted at 2026-08-07 22:48:36 on dist-test-slave-rgb1"
server_key: "973af8f4f6a612c2f77c6d796eec8064"
server_key_iv: "ba523c21184d363110ffd5d0b6f22d14"
server_key_version: "encryptionkey@0"
I20260807 22:49:18.555485 1162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:18.591248 1162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:18.593887 1162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:18.594077 1162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:18.598932 1162 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:43079
I20260807 22:49:18.598994 1233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:43079 every 8 connection(s)
I20260807 22:49:18.599437 1162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:18.601154 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 1162
I20260807 22:49:18.601342 1234 sys_catalog.cc:263] Verifying existing consensus state
I20260807 22:49:18.601490 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:33245
--local_ip_for_outbound_sockets=127.16.98.129
--tserver_master_addrs=127.16.98.190:43079
--webserver_port=33585
--webserver_interface=127.16.98.129
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:49:18.602443 1234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Bootstrap starting.
I20260807 22:49:18.613897 1234 log.cc:824] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:18.616786 1234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:49:18.617164 1234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Bootstrap complete.
I20260807 22:49:18.619877 1234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } }
I20260807 22:49:18.620165 1234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e3a23ef2354b1d89292ed4831f40c0, State: Initialized, Role: FOLLOWER
I20260807 22:49:18.620371 1234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } }
I20260807 22:49:18.620460 1234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:18.620524 1234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:18.620618 1234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 2 FOLLOWER]: Advancing to term 3
I20260807 22:49:18.621851 1234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } }
I20260807 22:49:18.621999 1234 leader_election.cc:302] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86e3a23ef2354b1d89292ed4831f40c0; no voters:
I20260807 22:49:18.622238 1234 leader_election.cc:288] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 22:49:18.622282 1238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 22:49:18.622431 1238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [term 3 LEADER]: Becoming Leader. State: Replica: 86e3a23ef2354b1d89292ed4831f40c0, State: Running, Role: LEADER
I20260807 22:49:18.622556 1234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:18.622534 1238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } }
I20260807 22:49:18.622970 1239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "86e3a23ef2354b1d89292ed4831f40c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } } }
I20260807 22:49:18.623070 1239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:18.623365 1240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86e3a23ef2354b1d89292ed4831f40c0. Latest consensus state: current_term: 3 leader_uuid: "86e3a23ef2354b1d89292ed4831f40c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e3a23ef2354b1d89292ed4831f40c0" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43079 } } }
I20260807 22:49:18.623483 1240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:18.623636 1246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:18.624658 1246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:18.624783 1246 catalog_manager.cc:1259] Loaded cluster ID: 75bcb32455bf41f49b13950f9d8a3352
I20260807 22:49:18.624832 1246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:49:18.625847 1246 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:18.626055 1246 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Loaded TSK: 0
I20260807 22:49:18.626116 1246 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 86e3a23ef2354b1d89292ed4831f40c0: Loaded TSK: 1
F20260807 22:49:18.626426 1246 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2836bb39ad google::LogMessage::Fail() at ??:0
@ 0x7f2836bb688a google::LogMessage::SendToLog() at ??:0
@ 0x7f2836bb34de google::LogMessage::Flush() at ??:0
@ 0x7f2836bb4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2839320104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f283931dd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f283eb0d1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f283eb0d48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f283eb0d594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f283ead3c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f283eaddad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2838a810d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2838a76954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f28398da6db start_thread at ??:0
@ 0x7f28356b161f clone at ??:0
W20260807 22:49:18.713377 1236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:18.713631 1236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:18.713675 1236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:18.714911 1236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:18.714977 1236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:18.715029 1236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:49:18.716482 1236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:33245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=33585
--tserver_master_addrs=127.16.98.190:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:18.716775 1236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:18.717023 1236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:18.722342 1279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:18.722272 1276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:18.722491 1236 server_base.cc:1034] running on GCE node
W20260807 22:49:18.722563 1277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:18.722783 1236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:18.723129 1236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:18.724308 1236 hybrid_clock.cc:648] HybridClock initialized: now 1786142958724283 us; error 44 us; skew 500 ppm
Aug 07 22:49:18 dist-test-slave-rgb1 krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142958, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:18.727173 1236 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:49:18.727699 1236 webserver.cc:460] Webserver started at http://127.16.98.129:33585/ using document root <none> and password file <none>
I20260807 22:49:18.727962 1236 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:18.728016 1236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:18.729537 1236 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:18.730176 1287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:18.730327 1236 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260807 22:49:18.730432 1236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "705e830ad7b74ed6b85e48ec392fb19a"
format_stamp: "Formatted at 2026-08-07 22:48:36 on dist-test-slave-rgb1"
server_key: "b3efa980fa077a4db385ac0a9acc4433"
server_key_iv: "24b1fdc10c2bd1c3b6895d33103976ce"
server_key_version: "encryptionkey@0"
I20260807 22:49:18.730798 1236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:18.748345 1236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:18.751137 1236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:18.751303 1236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:18.751564 1236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:18.751909 1236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:18.751962 1236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:18.752010 1236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:18.752034 1236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:18.759298 1236 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:33245
I20260807 22:49:18.759392 1404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:33245 every 8 connection(s)
I20260807 22:49:18.759768 1236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:49:18.768690 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 1236
I20260807 22:49:18.769045 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:44679
--local_ip_for_outbound_sockets=127.16.98.130
--tserver_master_addrs=127.16.98.190:43079
--webserver_port=43485
--webserver_interface=127.16.98.130
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:18.872362 1408 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:49:18.760676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:48421)
0807 22:49:18.760910 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:18.760960 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:18.761474 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:18.871801 (+110327us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":108,"threads_started":1}
W20260807 22:49:18.872807 1405 heartbeater.cc:644] Failed to heartbeat to 127.16.98.190:43079 (0 consecutive failures): Network error: Failed to ping master at 127.16.98.190:43079: Client connection negotiation failed: client connection to 127.16.98.190:43079: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 22:49:18.873337 1408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:18.873098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:37513)
0807 22:49:18.873160 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:18.873301 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260807 22:49:18.873818 1408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:18.873711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:42483)
0807 22:49:18.873761 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:18.873796 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260807 22:49:18.874047 1405 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 22:49:18.883695 1409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:18.883947 1409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:18.883977 1409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:18.885437 1409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:18.885504 1409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:18.885537 1409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:49:18.887256 1409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:44679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=43485
--tserver_master_addrs=127.16.98.190:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:18.887552 1409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:18.887813 1409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:18.893271 1416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:18.893306 1418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:18.893271 1415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:18.893479 1409 server_base.cc:1034] running on GCE node
I20260807 22:49:18.893764 1409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:18.893977 1409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:18.895131 1409 hybrid_clock.cc:648] HybridClock initialized: now 1786142958895112 us; error 34 us; skew 500 ppm
Aug 07 22:49:18 dist-test-slave-rgb1 krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142958, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:18.898082 1409 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:49:18.898553 1409 webserver.cc:460] Webserver started at http://127.16.98.130:43485/ using document root <none> and password file <none>
I20260807 22:49:18.898762 1409 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:18.898808 1409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:18.900295 1409 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:18.901078 1425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:18.901257 1409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 22:49:18.901353 1409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "4df16ca22bc248069ab8864eeab8d81a"
format_stamp: "Formatted at 2026-08-07 22:48:37 on dist-test-slave-rgb1"
server_key: "f432dc09089422c0bddf2d447d9501f9"
server_key_iv: "9fb42fbafb6c336eb123e6a9dcfff69e"
server_key_version: "encryptionkey@0"
I20260807 22:49:18.901727 1409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:18.929353 1409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:18.931931 1409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:18.932070 1409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:18.932299 1409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:18.932667 1409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:18.932713 1409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:18.932739 1409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:18.932755 1409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:18.939880 1409 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:44679
I20260807 22:49:18.939944 1543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:44679 every 8 connection(s)
I20260807 22:49:18.941689 1409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:49:18.943118 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:18.942568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:39001)
0807 22:49:18.942784 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:18.942942 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":94,"threads_started":1}
W20260807 22:49:18.943600 1544 heartbeater.cc:644] Failed to heartbeat to 127.16.98.190:43079 (0 consecutive failures): Network error: Failed to ping master at 127.16.98.190:43079: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
I20260807 22:49:18.943981 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:18.943837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:60797)
0807 22:49:18.943906 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:18.943955 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 22:49:18.944329 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:18.944247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:50117)
0807 22:49:18.944287 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:18.944311 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260807 22:49:18.944458 1544 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 22:49:18.946920 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 1409
I20260807 22:49:18.947332 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:42159
--local_ip_for_outbound_sockets=127.16.98.131
--tserver_master_addrs=127.16.98.190:43079
--webserver_port=36505
--webserver_interface=127.16.98.131
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:19.042217 1547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:19.042464 1547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:19.042495 1547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:19.043897 1547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:19.043956 1547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:19.043990 1547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:49:19.045742 1547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:42159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=36505
--tserver_master_addrs=127.16.98.190:43079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:19.046046 1547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:19.046320 1547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 22:49:19.052258 1547 server_base.cc:1034] running on GCE node
W20260807 22:49:19.052249 1553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:19.052407 1556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:19.052222 1554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:19.052771 1547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:19.053025 1547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:19.054203 1547 hybrid_clock.cc:648] HybridClock initialized: now 1786142959054190 us; error 33 us; skew 500 ppm
Aug 07 22:49:19 dist-test-slave-rgb1 krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142959, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:19.056991 1547 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:49:19.057488 1547 webserver.cc:460] Webserver started at http://127.16.98.131:36505/ using document root <none> and password file <none>
I20260807 22:49:19.057705 1547 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:19.057751 1547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:19.059202 1547 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:19.059950 1565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:19.060184 1547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:19.060272 1547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "457d288fc4724700944d815621854da1"
format_stamp: "Formatted at 2026-08-07 22:48:37 on dist-test-slave-rgb1"
server_key: "32fdeff36089f73873ac112bddb23976"
server_key_iv: "0bf197a3d1099eee7d1f1c9502bcafc8"
server_key_version: "encryptionkey@0"
I20260807 22:49:19.060770 1547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:19.071817 1547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:19.074836 1547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:19.075068 1547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:19.075301 1547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:19.075779 1547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:19.075829 1547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:19.075855 1547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:19.075868 1547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:19.083040 1547 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:42159
I20260807 22:49:19.083089 1682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:42159 every 8 connection(s)
I20260807 22:49:19.083504 1547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:49:19.084053 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 1547
I20260807 22:49:19.084824 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:19.084292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:36307)
0807 22:49:19.084478 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:19.084610 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
W20260807 22:49:19.085260 1683 heartbeater.cc:644] Failed to heartbeat to 127.16.98.190:43079 (0 consecutive failures): Network error: Failed to ping master at 127.16.98.190:43079: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
I20260807 22:49:19.085626 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:19.085476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:57643)
0807 22:49:19.085562 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:19.085598 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260807 22:49:19.085997 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:19.085895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:50159)
0807 22:49:19.085937 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:19.085978 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260807 22:49:19.086227 1683 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 22:49:19.874887 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:19.874545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:41043)
0807 22:49:19.874745 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:19.874796 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260807 22:49:19.945140 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:19.944849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:55745)
0807 22:49:19.945033 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:19.945080 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260807 22:49:20.086881 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:20.086618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:54223)
0807 22:49:20.086771 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:20.086816 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260807 22:49:20.875823 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:20.875567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:59519)
0807 22:49:20.875730 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:20.875772 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20260807 22:49:20.945960 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:20.945688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:58367)
0807 22:49:20.945857 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:20.945910 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":73,"threads_started":1}
I20260807 22:49:21.087872 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:21.087612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:46167)
0807 22:49:21.087783 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:21.087820 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
I20260807 22:49:21.876889 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:21.876541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:37661)
0807 22:49:21.876776 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:21.876831 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":138,"threads_started":1}
I20260807 22:49:21.946774 1694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:21.946503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:34643)
0807 22:49:21.946653 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:21.946700 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260807 22:49:22.088831 1695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:22.088501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:37687)
0807 22:49:22.088719 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:22.088780 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":124,"threads_started":1}
I20260807 22:49:22.877771 1697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:22.877500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:37131)
0807 22:49:22.877667 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:22.877715 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":72,"threads_started":1}
I20260807 22:49:22.947698 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:22.947444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:49387)
0807 22:49:22.947589 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:22.947648 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260807 22:49:23.089663 1699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:23.089399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:33937)
0807 22:49:23.089555 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:23.089612 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260807 22:49:23.878918 1700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:23.878605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:57475)
0807 22:49:23.878797 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:23.878847 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
I20260807 22:49:23.948491 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:23.948227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:56839)
0807 22:49:23.948371 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:23.948438 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260807 22:49:24.090524 1702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:24.090252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:54647)
0807 22:49:24.090411 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:24.090471 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260807 22:49:24.880070 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:24.879727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:57837)
0807 22:49:24.879928 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:24.879979 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":136,"threads_started":1}
I20260807 22:49:24.949465 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:24.949191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:40451)
0807 22:49:24.949363 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:24.949414 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":72,"threads_started":1}
I20260807 22:49:25.091457 1705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:25.091123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:59919)
0807 22:49:25.091351 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:25.091402 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":110,"threads_started":1}
I20260807 22:49:25.881035 1706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:25.880735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:41649)
0807 22:49:25.880931 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:25.880979 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":70,"threads_started":1}
I20260807 22:49:25.950531 1707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:25.950243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:32983)
0807 22:49:25.950421 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:25.950462 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":107,"threads_started":1}
I20260807 22:49:26.092406 1708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:26.092115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:56815)
0807 22:49:26.092277 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:26.092348 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":72,"threads_started":1}
I20260807 22:49:26.881901 1709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:26.881640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:45493)
0807 22:49:26.881804 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:26.881849 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
I20260807 22:49:26.951509 1710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:26.951240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:38497)
0807 22:49:26.951402 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:26.951440 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260807 22:49:27.093353 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:27.093081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:41059)
0807 22:49:27.093252 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:27.093298 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":74,"threads_started":1}
I20260807 22:49:27.882920 1712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:27.882619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:52089)
0807 22:49:27.882799 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:27.882861 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":71,"threads_started":1}
I20260807 22:49:27.952502 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:27.952251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:50101)
0807 22:49:27.952390 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:27.952431 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260807 22:49:28.094226 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:28.093935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:44913)
0807 22:49:28.094127 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:28.094175 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20260807 22:49:28.884373 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:28.883887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:49061)
0807 22:49:28.884241 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:28.884304 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":214,"threads_started":1}
I20260807 22:49:28.953428 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:28.953128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:54299)
0807 22:49:28.953291 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:28.953346 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260807 22:49:29.095132 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:29.094825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:58995)
0807 22:49:29.095031 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:29.095079 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":88,"threads_started":1}
I20260807 22:49:29.885356 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:29.885023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:43687)
0807 22:49:29.885209 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:29.885265 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260807 22:49:29.954382 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:29.954065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:51747)
0807 22:49:29.954283 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:29.954329 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
I20260807 22:49:30.096246 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:30.095883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:41231)
0807 22:49:30.096135 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:30.096185 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":102,"threads_started":1}
I20260807 22:49:30.886253 1721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:30.885946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:44999)
0807 22:49:30.886132 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:30.886182 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":83,"threads_started":1}
I20260807 22:49:30.955299 1722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:30.955036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:53195)
0807 22:49:30.955200 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:30.955250 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
I20260807 22:49:31.097257 1723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:31.096996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:41135)
0807 22:49:31.097155 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:31.097205 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260807 22:49:31.887159 1724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:31.886837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:38367)
0807 22:49:31.887013 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:31.887095 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
I20260807 22:49:31.956303 1725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:31.955991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:59697)
0807 22:49:31.956208 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:31.956255 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
I20260807 22:49:32.098178 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:32.097890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:52249)
0807 22:49:32.098060 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:32.098112 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20260807 22:49:32.888083 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:32.887757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:56577)
0807 22:49:32.887981 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:32.888030 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
I20260807 22:49:32.957476 1728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:32.957130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:43847)
0807 22:49:32.957363 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:32.957419 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":142,"threads_started":1}
I20260807 22:49:33.099043 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:33.098739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:59441)
0807 22:49:33.098942 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:33.098984 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
I20260807 22:49:33.889000 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:33.888715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.129:54149)
0807 22:49:33.888914 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:33.888951 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260807 22:49:33.958432 1731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:33.958151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.130:53693)
0807 22:49:33.958331 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:33.958380 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
I20260807 22:49:34.090920 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 1236
I20260807 22:49:34.096334 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 1409
I20260807 22:49:34.099970 1732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:34.099708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43079 (local address 127.16.98.131:38211)
0807 22:49:34.099876 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:34.099919 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.190:43079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
I20260807 22:49:34.102569 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 1547
2026-08-07T22:49:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58737 ms)
[----------] 30 tests from SecurityITest (257693 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:49:34 dist-test-slave-rgb1 krb5kdc[1736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:49:34 dist-test-slave-rgb1 krb5kdc[1736](info): set up 2 sockets
Aug 07 22:49:34 dist-test-slave-rgb1 krb5kdc[1736](info): commencing operation
krb5kdc: starting...
W20260807 22:49:35.222563 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:49:35 dist-test-slave-rgb1 krb5kdc[1736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:49:35.373241 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:33403
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:33403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:35.462658 1749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:35.462893 1749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:35.462917 1749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:35.464015 1749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:35.464054 1749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:35.464068 1749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:35.464082 1749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:35.464094 1749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:35.465867 1749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:33403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:33403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:35.466181 1749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:35.466483 1749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:35.472425 1757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:35.472466 1749 server_base.cc:1034] running on GCE node
W20260807 22:49:35.472431 1755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:35.472425 1754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:35.473032 1749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:35.473354 1749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:35.474561 1749 hybrid_clock.cc:648] HybridClock initialized: now 1786142975474517 us; error 55 us; skew 500 ppm
Aug 07 22:49:35 dist-test-slave-rgb1 krb5kdc[1736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142975, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:35.477676 1749 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:49:35.478173 1749 webserver.cc:460] Webserver started at http://127.16.98.190:42471/ using document root <none> and password file <none>
I20260807 22:49:35.478420 1749 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:35.478490 1749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:35.478647 1749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:35.479933 1749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "8c9ed11e4ae2413fa0a33b3d206e5332"
format_stamp: "Formatted at 2026-08-07 22:49:35 on dist-test-slave-rgb1"
server_key: "8dea137c87b9e5f29d62338138940494"
server_key_iv: "c2c901bf18abb6bdfc079a46dd81b8cb"
server_key_version: "encryptionkey@0"
I20260807 22:49:35.480371 1749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "8c9ed11e4ae2413fa0a33b3d206e5332"
format_stamp: "Formatted at 2026-08-07 22:49:35 on dist-test-slave-rgb1"
server_key: "8dea137c87b9e5f29d62338138940494"
server_key_iv: "c2c901bf18abb6bdfc079a46dd81b8cb"
server_key_version: "encryptionkey@0"
I20260807 22:49:35.481925 1749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:49:35.482800 1764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:35.483050 1749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:49:35.483162 1749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "8c9ed11e4ae2413fa0a33b3d206e5332"
format_stamp: "Formatted at 2026-08-07 22:49:35 on dist-test-slave-rgb1"
server_key: "8dea137c87b9e5f29d62338138940494"
server_key_iv: "c2c901bf18abb6bdfc079a46dd81b8cb"
server_key_version: "encryptionkey@0"
I20260807 22:49:35.483263 1749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:35.495684 1749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:35.498188 1749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:35.498370 1749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:35.502830 1749 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:33403
I20260807 22:49:35.502892 1818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:33403 every 8 connection(s)
I20260807 22:49:35.503285 1749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:35.504227 1819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:35.506681 1819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332: Bootstrap starting.
I20260807 22:49:35.507252 1819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:35.507509 1819 log.cc:824] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:35.508234 1819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332: No bootstrap required, opened a new log
I20260807 22:49:35.509461 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 1749
I20260807 22:49:35.509624 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:35.509853 16778 external_mini_cluster.cc:1428] Setting key a7c03956ad93cfd8b74819ab12be2ebe
I20260807 22:49:35.510179 1819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9ed11e4ae2413fa0a33b3d206e5332" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 33403 } }
I20260807 22:49:35.510336 1819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:35.510360 1819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9ed11e4ae2413fa0a33b3d206e5332, State: Initialized, Role: FOLLOWER
I20260807 22:49:35.510499 1819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9ed11e4ae2413fa0a33b3d206e5332" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 33403 } }
I20260807 22:49:35.510605 1819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:35.510651 1819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:35.510711 1819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:35.512135 1819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9ed11e4ae2413fa0a33b3d206e5332" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 33403 } }
I20260807 22:49:35.512266 1819 leader_election.cc:302] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c9ed11e4ae2413fa0a33b3d206e5332; no voters:
I20260807 22:49:35.512492 1819 leader_election.cc:288] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:35.512533 1824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:35.512665 1824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [term 1 LEADER]: Becoming Leader. State: Replica: 8c9ed11e4ae2413fa0a33b3d206e5332, State: Running, Role: LEADER
I20260807 22:49:35.512816 1824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9ed11e4ae2413fa0a33b3d206e5332" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 33403 } }
I20260807 22:49:35.512815 1819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:35.513257 1826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c9ed11e4ae2413fa0a33b3d206e5332" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9ed11e4ae2413fa0a33b3d206e5332" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 33403 } } }
I20260807 22:49:35.513363 1826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:35.513258 1825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c9ed11e4ae2413fa0a33b3d206e5332. Latest consensus state: current_term: 1 leader_uuid: "8c9ed11e4ae2413fa0a33b3d206e5332" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9ed11e4ae2413fa0a33b3d206e5332" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 33403 } } }
I20260807 22:49:35.513561 1825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:35.513586 1828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:35.514164 1828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:35.517241 1828 catalog_manager.cc:1347] Generated new cluster ID: 4de2bfa66c5e4eb8a0f1a4ec14cc0b20
I20260807 22:49:35.517481 1828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 22:49:35 dist-test-slave-rgb1 krb5kdc[1736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:49:35.523898 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:35.510792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43074 (local address 127.16.98.190:33403)
0807 22:49:35.511000 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:35.511009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:35.511058 (+ 49us) server_negotiation.cc:407] Connection header received
0807 22:49:35.511166 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:35.511173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:35.511232 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:35.511342 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:35.512237 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:35.513766 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:35.515079 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:35.515777 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:35.515935 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:35.519456 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:35.519461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:35.519462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:35.519566 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:35.521854 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:35.522296 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:35.522299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:35.522299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:35.522436 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:35.522619 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:35.522619 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:35.522620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:35.522927 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:35.523449 (+ 522us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:35.523522 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 22:49:35.523670 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260807 22:49:35.544407 1828 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:35.545013 1828 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:35.548233 1828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332: Generated new TSK 0
I20260807 22:49:35.548441 1828 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:49:35.585687 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:33403
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:35.673139 1847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:35.673384 1847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:35.673424 1847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:35.674597 1847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:35.674665 1847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:35.674712 1847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:49:35.676663 1847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:33403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:35.677009 1847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:35.677310 1847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:35.683475 1855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:35.683501 1853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:35.683650 1847 server_base.cc:1034] running on GCE node
W20260807 22:49:35.683574 1852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:35.683969 1847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:35.684307 1847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:35.685496 1847 hybrid_clock.cc:648] HybridClock initialized: now 1786142975685477 us; error 58 us; skew 500 ppm
Aug 07 22:49:35 dist-test-slave-rgb1 krb5kdc[1736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142975, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:35.688745 1847 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:49:35.689232 1847 webserver.cc:460] Webserver started at http://127.16.98.129:39717/ using document root <none> and password file <none>
I20260807 22:49:35.689498 1847 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:35.689569 1847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:35.689743 1847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:35.690904 1847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "5a92ae26c0bd42a880fb07b575a13f3f"
format_stamp: "Formatted at 2026-08-07 22:49:35 on dist-test-slave-rgb1"
server_key: "cb32eeb03bd7b170ed28a8013b4c4b3f"
server_key_iv: "ccb8a651dc7151321b74dcda9e97d7e3"
server_key_version: "encryptionkey@0"
I20260807 22:49:35.691277 1847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "5a92ae26c0bd42a880fb07b575a13f3f"
format_stamp: "Formatted at 2026-08-07 22:49:35 on dist-test-slave-rgb1"
server_key: "cb32eeb03bd7b170ed28a8013b4c4b3f"
server_key_iv: "ccb8a651dc7151321b74dcda9e97d7e3"
server_key_version: "encryptionkey@0"
I20260807 22:49:35.692802 1847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:49:35.693763 1862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:35.694027 1847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:35.694144 1847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "5a92ae26c0bd42a880fb07b575a13f3f"
format_stamp: "Formatted at 2026-08-07 22:49:35 on dist-test-slave-rgb1"
server_key: "cb32eeb03bd7b170ed28a8013b4c4b3f"
server_key_iv: "ccb8a651dc7151321b74dcda9e97d7e3"
server_key_version: "encryptionkey@0"
I20260807 22:49:35.694226 1847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:35.708149 1847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:35.710414 1847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:35.710592 1847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:35.710841 1847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:35.711193 1847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:35.711266 1847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:35.711311 1847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:35.711344 1847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:35.717796 1847 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:40437
I20260807 22:49:35.717850 1977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:40437 every 8 connection(s)
I20260807 22:49:35.718181 1847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:49:35.723155 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 1847
I20260807 22:49:35.723264 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:49:35.723506 16778 external_mini_cluster.cc:1428] Setting key e118c49a11fd9b5ac702822b11666115
Aug 07 22:49:35 dist-test-slave-rgb1 krb5kdc[1736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142975, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:49:35.729497 1980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:35.718876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:33403 (local address 127.16.98.129:53093)
0807 22:49:35.719052 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:35.719097 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:35.719587 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:35.719837 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:35.719840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:35.720419 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:35.720650 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:35.720655 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:35.721434 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:35.721435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:35.722413 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:35.722416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:35.722965 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:35.722967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:35.723091 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:35.723905 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:49:35.723917 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:35.725691 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:35.727861 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:35.727870 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:35.727872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:35.728167 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:35.728439 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:35.728440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:35.728440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:35.728545 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:35.728924 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:35.728930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:35.729101 (+ 171us) client_negotiation.cc:769] Sending connection context
0807 22:49:35.729186 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 22:49:35.729319 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260807 22:49:35.729563 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:35.718930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:53093 (local address 127.16.98.190:33403)
0807 22:49:35.718991 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:35.718994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:35.719210 (+ 216us) server_negotiation.cc:407] Connection header received
0807 22:49:35.719688 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:35.719689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:35.719706 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:35.719785 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:35.720770 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:35.721305 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:35.722528 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:35.722771 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:35.722896 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:35.725805 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:35.725807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:35.725808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:35.725818 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:35.727762 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:35.728287 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:35.728288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:35.728289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:35.728364 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:35.728664 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:35.728665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:35.728665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:35.728849 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:35.728931 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:35.729269 (+ 338us) server_negotiation.cc:299] Negotiation successful
0807 22:49:35.729394 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260807 22:49:35.730393 1978 heartbeater.cc:347] Connected to a master server at 127.16.98.190:33403
I20260807 22:49:35.730535 1978 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:35.730762 1978 heartbeater.cc:511] Master 127.16.98.190:33403 requested a full tablet report, sending...
I20260807 22:49:35.731415 1781 ts_manager.cc:194] Registered new tserver with Master: 5a92ae26c0bd42a880fb07b575a13f3f (127.16.98.129:40437)
I20260807 22:49:35.732165 1781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:53093
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:49:35.757095 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:33403
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:35.845186 1983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:35.845408 1983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:35.845434 1983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:35.846644 1983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:35.846693 1983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:35.846724 1983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:49:35.848306 1983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:33403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:35.848654 1983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:35.848924 1983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:35.855001 1993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:35.855007 1990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:35.855012 1991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:35.855341 1983 server_base.cc:1034] running on GCE node
I20260807 22:49:35.855614 1983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:35.855842 1983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:35.857012 1983 hybrid_clock.cc:648] HybridClock initialized: now 1786142975856988 us; error 88 us; skew 500 ppm
Aug 07 22:49:35 dist-test-slave-rgb1 krb5kdc[1736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142975, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:35.860600 1983 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:49:35.861176 1983 webserver.cc:460] Webserver started at http://127.16.98.130:35077/ using document root <none> and password file <none>
I20260807 22:49:35.861426 1983 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:35.861496 1983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:35.861645 1983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:35.862773 1983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "760ca0f4bdea40899dbac334f6ee2324"
format_stamp: "Formatted at 2026-08-07 22:49:35 on dist-test-slave-rgb1"
server_key: "d026c96895e7bcf2ce3b9cf4ef79a0cf"
server_key_iv: "beda3a105386487c4fb325ce3e8b7b86"
server_key_version: "encryptionkey@0"
I20260807 22:49:35.863157 1983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "760ca0f4bdea40899dbac334f6ee2324"
format_stamp: "Formatted at 2026-08-07 22:49:35 on dist-test-slave-rgb1"
server_key: "d026c96895e7bcf2ce3b9cf4ef79a0cf"
server_key_iv: "beda3a105386487c4fb325ce3e8b7b86"
server_key_version: "encryptionkey@0"
I20260807 22:49:35.864676 1983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:49:35.865551 2000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:35.865955 1983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:35.866093 1983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "760ca0f4bdea40899dbac334f6ee2324"
format_stamp: "Formatted at 2026-08-07 22:49:35 on dist-test-slave-rgb1"
server_key: "d026c96895e7bcf2ce3b9cf4ef79a0cf"
server_key_iv: "beda3a105386487c4fb325ce3e8b7b86"
server_key_version: "encryptionkey@0"
I20260807 22:49:35.866194 1983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:35.887646 1983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:35.890422 1983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:35.890631 1983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:35.890929 1983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:35.891319 1983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:35.891382 1983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:35.891426 1983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:35.891479 1983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:35.898495 1983 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:45473
I20260807 22:49:35.898550 2114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:45473 every 8 connection(s)
I20260807 22:49:35.898958 1983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:49:35.904088 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 1983
I20260807 22:49:35.904237 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:49:35.904452 16778 external_mini_cluster.cc:1428] Setting key fa0ce342bfcd96d8e411b6dec5538ae5
Aug 07 22:49:35 dist-test-slave-rgb1 krb5kdc[1736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142975, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:49:35.911230 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:35.900001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:43709 (local address 127.16.98.190:33403)
0807 22:49:35.900124 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:35.900127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:35.900434 (+ 307us) server_negotiation.cc:407] Connection header received
0807 22:49:35.901021 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:35.901022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:35.901034 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:35.901098 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:35.902279 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:35.902792 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:35.904263 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:35.904623 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:35.904719 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:35.907638 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:35.907640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:35.907640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:35.907650 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:35.909503 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:35.910111 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:35.910112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:35.910113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:35.910182 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:35.910478 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:35.910480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:35.910480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:35.910618 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:35.910705 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:35.911022 (+ 317us) server_negotiation.cc:299] Negotiation successful
0807 22:49:35.911126 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 22:49:35.911469 2117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:35.900031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:33403 (local address 127.16.98.130:43709)
0807 22:49:35.900287 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:35.900328 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:35.900860 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:35.901223 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:35.901227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:35.901933 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:35.902160 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:35.902165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:35.902933 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:35.902934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:35.904125 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:35.904129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:35.904736 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:35.904737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:35.904900 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:35.905565 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:49:35.905577 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:35.907529 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:35.909623 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:35.909632 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:35.909635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:35.909988 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:35.910259 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:35.910260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:35.910261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:35.910393 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:35.910696 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:35.910703 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:35.910862 (+ 159us) client_negotiation.cc:769] Sending connection context
0807 22:49:35.910985 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 22:49:35.911178 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":121,"threads_started":1}
I20260807 22:49:35.911998 2115 heartbeater.cc:347] Connected to a master server at 127.16.98.190:33403
I20260807 22:49:35.912118 2115 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:35.912328 2115 heartbeater.cc:511] Master 127.16.98.190:33403 requested a full tablet report, sending...
I20260807 22:49:35.912879 1781 ts_manager.cc:194] Registered new tserver with Master: 760ca0f4bdea40899dbac334f6ee2324 (127.16.98.130:45473)
I20260807 22:49:35.913385 1781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:43709
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:49:35.936249 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:33403
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:36.032130 2120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:36.032369 2120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:36.032397 2120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:36.033774 2120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:36.033826 2120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:36.033857 2120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:49:36.035619 2120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:33403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:36.035893 2120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:36.036163 2120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:36.041733 2129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:36.041857 2127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:36.041977 2120 server_base.cc:1034] running on GCE node
W20260807 22:49:36.041733 2125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:36.042294 2120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:36.042564 2120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:36.043741 2120 hybrid_clock.cc:648] HybridClock initialized: now 1786142976043735 us; error 39 us; skew 500 ppm
Aug 07 22:49:36 dist-test-slave-rgb1 krb5kdc[1736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142976, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:36.046577 2120 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:49:36.047101 2120 webserver.cc:460] Webserver started at http://127.16.98.131:45491/ using document root <none> and password file <none>
I20260807 22:49:36.047359 2120 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:36.047434 2120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:36.047603 2120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:36.048910 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "990e083672fb44a1b05a5e7747bea695"
format_stamp: "Formatted at 2026-08-07 22:49:36 on dist-test-slave-rgb1"
server_key: "9ab755e48defaf7f5fdd7632451d0815"
server_key_iv: "45f0f5b14ae608a246e5d16e4476741f"
server_key_version: "encryptionkey@0"
I20260807 22:49:36.049312 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "990e083672fb44a1b05a5e7747bea695"
format_stamp: "Formatted at 2026-08-07 22:49:36 on dist-test-slave-rgb1"
server_key: "9ab755e48defaf7f5fdd7632451d0815"
server_key_iv: "45f0f5b14ae608a246e5d16e4476741f"
server_key_version: "encryptionkey@0"
I20260807 22:49:36.050791 2120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:49:36.051680 2136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:36.052021 2120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:36.052156 2120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "990e083672fb44a1b05a5e7747bea695"
format_stamp: "Formatted at 2026-08-07 22:49:36 on dist-test-slave-rgb1"
server_key: "9ab755e48defaf7f5fdd7632451d0815"
server_key_iv: "45f0f5b14ae608a246e5d16e4476741f"
server_key_version: "encryptionkey@0"
I20260807 22:49:36.052237 2120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:36.068878 2120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:36.071640 2120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:36.071930 2120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:36.072222 2120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:36.072638 2120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:36.072696 2120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:36.072741 2120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:36.072767 2120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:36.079315 2120 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:33673
I20260807 22:49:36.079408 2253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:33673 every 8 connection(s)
I20260807 22:49:36.079736 2120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:49:36.083118 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 2120
I20260807 22:49:36.083256 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:49:36.083489 16778 external_mini_cluster.cc:1428] Setting key b09d7fcea7c5855575f75c186f37223f
Aug 07 22:49:36 dist-test-slave-rgb1 krb5kdc[1736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142976, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:49:36.091044 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.080618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:42187 (local address 127.16.98.190:33403)
0807 22:49:36.080678 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:36.080681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:36.080847 (+ 166us) server_negotiation.cc:407] Connection header received
0807 22:49:36.081645 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:36.081646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:36.081664 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:36.081738 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:36.083151 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.084103 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.085328 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.085566 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.085651 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.088136 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:36.088138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:36.088138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:36.088146 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:36.089575 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:36.090039 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:36.090040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:36.090041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:36.090103 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:36.090364 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:36.090365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:36.090366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:36.090490 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:36.090574 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:36.090863 (+ 289us) server_negotiation.cc:299] Negotiation successful
0807 22:49:36.090949 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 22:49:36.091269 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.080479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:33403 (local address 127.16.98.131:42187)
0807 22:49:36.080703 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:36.080745 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:36.081437 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:36.081842 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:36.081847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:36.082647 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:36.083026 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.083034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.084196 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.084197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.085210 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.085213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.085658 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.085659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.085822 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.086435 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:49:36.086443 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:36.087925 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:36.089683 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:36.089691 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:36.089693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:36.089938 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:36.090174 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:36.090175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:36.090175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:36.090292 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:36.090580 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:36.090588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:36.090731 (+ 143us) client_negotiation.cc:769] Sending connection context
0807 22:49:36.090842 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 22:49:36.091021 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":126,"threads_started":1}
I20260807 22:49:36.091804 2254 heartbeater.cc:347] Connected to a master server at 127.16.98.190:33403
I20260807 22:49:36.091902 2254 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:36.092064 2254 heartbeater.cc:511] Master 127.16.98.190:33403 requested a full tablet report, sending...
I20260807 22:49:36.092432 1781 ts_manager.cc:194] Registered new tserver with Master: 990e083672fb44a1b05a5e7747bea695 (127.16.98.131:33673)
I20260807 22:49:36.092834 1781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:42187
I20260807 22:49:36.095377 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:49:36.103288 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.096988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.16.98.190:33403)
0807 22:49:36.097063 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:36.097065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:36.097176 (+ 111us) server_negotiation.cc:407] Connection header received
0807 22:49:36.097225 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:36.097226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:36.097234 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:36.097280 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:36.097878 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.098378 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.099185 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.099397 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.099466 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.100761 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:36.100763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:36.100763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:36.100769 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:36.102153 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:36.102491 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:36.102492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:36.102493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:36.102553 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:36.102734 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:36.102734 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:36.102735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:36.102875 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:36.102954 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:36.103074 (+ 120us) server_negotiation.cc:299] Negotiation successful
0807 22:49:36.103144 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 22:49:36.105125 1781 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:49:36.106828 1781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:49:36.113978 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.109978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.16.98.129:40437)
0807 22:49:36.110185 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:36.110189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:36.110202 (+ 13us) server_negotiation.cc:407] Connection header received
0807 22:49:36.110244 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:36.110246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:36.110261 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:36.110523 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:36.110633 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.111378 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.112863 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.113534 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.113594 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.113638 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:36.113830 (+ 192us) server_negotiation.cc:299] Negotiation successful
0807 22:49:36.113881 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":99,"threads_started":1}
I20260807 22:49:36.114002 2268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.109755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:33673 (local address 127.0.0.1:56740)
0807 22:49:36.109871 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:36.109881 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:36.109925 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:36.110619 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:36.110620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:36.110623 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:36.110659 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.110661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.111901 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.111902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.112941 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.112945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.113721 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.113723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.113790 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.113795 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:49:36.113852 (+ 57us) client_negotiation.cc:240] Negotiation successful
0807 22:49:36.113918 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260807 22:49:36.114013 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.109462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:40437 (local address 127.0.0.1:53404)
0807 22:49:36.109638 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:36.109680 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:36.109745 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:36.110370 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:36.110371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:36.110373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:36.110429 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.110432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.111488 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.111489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.112775 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.112778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.113615 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.113616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.113723 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.113740 (+ 17us) client_negotiation.cc:769] Sending connection context
0807 22:49:36.113812 (+ 72us) client_negotiation.cc:240] Negotiation successful
0807 22:49:36.113903 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":28,"thread_start_us":119,"threads_started":1}
I20260807 22:49:36.114758 2271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.110052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.16.98.131:33673)
0807 22:49:36.110359 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:36.110363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:36.110373 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:49:36.110411 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:36.110413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:36.110427 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:36.110597 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:36.110728 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.111787 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.113044 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.113628 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.113708 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.113766 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:36.114621 (+ 855us) server_negotiation.cc:299] Negotiation successful
0807 22:49:36.114669 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":120,"threads_started":1}
I20260807 22:49:36.115072 2267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.109451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:45473 (local address 127.0.0.1:56502)
0807 22:49:36.109638 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:36.109679 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:36.109743 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:36.110345 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:36.110348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:36.110351 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:36.110426 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.110429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.111833 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.111834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.112958 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.112960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.114748 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.114749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.114815 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.114820 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:49:36.114875 (+ 55us) client_negotiation.cc:240] Negotiation successful
0807 22:49:36.114918 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":1,"thread_start_us":136,"threads_started":1}
I20260807 22:49:36.115178 2269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.109734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56502 (local address 127.16.98.130:45473)
0807 22:49:36.110058 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:36.110064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:36.110075 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:49:36.110119 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:36.110121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:36.110137 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:36.110227 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:36.110519 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.111721 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.113809 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.114590 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.114657 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.114714 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:36.114892 (+ 178us) server_negotiation.cc:299] Negotiation successful
0807 22:49:36.114932 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":104,"threads_started":1}
I20260807 22:49:36.115072 1911 tablet_service.cc:1467] Processing CreateTablet for tablet f601f2b2428a42898407b304fb105cd1 (DEFAULT_TABLE table=test-table [id=bd78dd3a22e04388a09bce53ae213daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:36.115463 1911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f601f2b2428a42898407b304fb105cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:36.115554 2186 tablet_service.cc:1467] Processing CreateTablet for tablet f601f2b2428a42898407b304fb105cd1 (DEFAULT_TABLE table=test-table [id=bd78dd3a22e04388a09bce53ae213daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:36.115857 2186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f601f2b2428a42898407b304fb105cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:36.116015 2048 tablet_service.cc:1467] Processing CreateTablet for tablet f601f2b2428a42898407b304fb105cd1 (DEFAULT_TABLE table=test-table [id=bd78dd3a22e04388a09bce53ae213daa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:36.116254 2048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f601f2b2428a42898407b304fb105cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:36.119130 2272 tablet_bootstrap.cc:492] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324: Bootstrap starting.
I20260807 22:49:36.119652 2274 tablet_bootstrap.cc:492] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695: Bootstrap starting.
I20260807 22:49:36.119997 2272 tablet_bootstrap.cc:654] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:36.120288 2274 tablet_bootstrap.cc:654] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:36.120720 2273 tablet_bootstrap.cc:492] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f: Bootstrap starting.
I20260807 22:49:36.120868 2274 log.cc:824] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:36.120783 2272 log.cc:824] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:36.121449 2273 tablet_bootstrap.cc:654] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:36.121824 2274 tablet_bootstrap.cc:492] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695: No bootstrap required, opened a new log
I20260807 22:49:36.121915 2274 ts_tablet_manager.cc:1397] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:49:36.122046 2273 log.cc:824] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:36.122885 2273 tablet_bootstrap.cc:492] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f: No bootstrap required, opened a new log
I20260807 22:49:36.122962 2273 ts_tablet_manager.cc:1397] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:49:36.123978 2274 raft_consensus.cc:348] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760ca0f4bdea40899dbac334f6ee2324" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45473 } } peers { permanent_uuid: "5a92ae26c0bd42a880fb07b575a13f3f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40437 } } peers { permanent_uuid: "990e083672fb44a1b05a5e7747bea695" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 33673 } }
I20260807 22:49:36.124174 2274 raft_consensus.cc:374] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:36.124215 2274 raft_consensus.cc:729] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 990e083672fb44a1b05a5e7747bea695, State: Initialized, Role: FOLLOWER
I20260807 22:49:36.124332 2274 consensus_queue.cc:260] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760ca0f4bdea40899dbac334f6ee2324" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45473 } } peers { permanent_uuid: "5a92ae26c0bd42a880fb07b575a13f3f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40437 } } peers { permanent_uuid: "990e083672fb44a1b05a5e7747bea695" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 33673 } }
I20260807 22:49:36.124671 2274 ts_tablet_manager.cc:1428] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:49:36.124730 2254 heartbeater.cc:503] Master 127.16.98.190:33403 was elected leader, sending a full tablet report...
I20260807 22:49:36.125195 2272 tablet_bootstrap.cc:492] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324: No bootstrap required, opened a new log
I20260807 22:49:36.125108 2273 raft_consensus.cc:348] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760ca0f4bdea40899dbac334f6ee2324" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45473 } } peers { permanent_uuid: "5a92ae26c0bd42a880fb07b575a13f3f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40437 } } peers { permanent_uuid: "990e083672fb44a1b05a5e7747bea695" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 33673 } }
I20260807 22:49:36.125324 2272 ts_tablet_manager.cc:1397] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260807 22:49:36.125367 2273 raft_consensus.cc:374] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:36.125465 2273 raft_consensus.cc:729] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a92ae26c0bd42a880fb07b575a13f3f, State: Initialized, Role: FOLLOWER
I20260807 22:49:36.125640 2273 consensus_queue.cc:260] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760ca0f4bdea40899dbac334f6ee2324" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45473 } } peers { permanent_uuid: "5a92ae26c0bd42a880fb07b575a13f3f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40437 } } peers { permanent_uuid: "990e083672fb44a1b05a5e7747bea695" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 33673 } }
I20260807 22:49:36.126039 2273 ts_tablet_manager.cc:1428] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:49:36.126106 1978 heartbeater.cc:503] Master 127.16.98.190:33403 was elected leader, sending a full tablet report...
I20260807 22:49:36.127422 2272 raft_consensus.cc:348] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760ca0f4bdea40899dbac334f6ee2324" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45473 } } peers { permanent_uuid: "5a92ae26c0bd42a880fb07b575a13f3f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40437 } } peers { permanent_uuid: "990e083672fb44a1b05a5e7747bea695" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 33673 } }
I20260807 22:49:36.127573 2272 raft_consensus.cc:374] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:36.127599 2272 raft_consensus.cc:729] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 760ca0f4bdea40899dbac334f6ee2324, State: Initialized, Role: FOLLOWER
I20260807 22:49:36.127707 2272 consensus_queue.cc:260] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760ca0f4bdea40899dbac334f6ee2324" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45473 } } peers { permanent_uuid: "5a92ae26c0bd42a880fb07b575a13f3f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40437 } } peers { permanent_uuid: "990e083672fb44a1b05a5e7747bea695" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 33673 } }
I20260807 22:49:36.128014 2272 ts_tablet_manager.cc:1428] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260807 22:49:36.128026 2115 heartbeater.cc:503] Master 127.16.98.190:33403 was elected leader, sending a full tablet report...
W20260807 22:49:36.149672 2116 tablet.cc:2369] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:49:36.218966 1979 tablet.cc:2369] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:49:36.330389 2255 tablet.cc:2369] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:49:36.349367 2278 raft_consensus.cc:482] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:49:36.349511 2278 raft_consensus.cc:504] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760ca0f4bdea40899dbac334f6ee2324" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45473 } } peers { permanent_uuid: "5a92ae26c0bd42a880fb07b575a13f3f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40437 } } peers { permanent_uuid: "990e083672fb44a1b05a5e7747bea695" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 33673 } }
I20260807 22:49:36.349877 2278 leader_election.cc:288] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 760ca0f4bdea40899dbac334f6ee2324 (127.16.98.130:45473), 5a92ae26c0bd42a880fb07b575a13f3f (127.16.98.129:40437)
I20260807 22:49:36.353077 2269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.350293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:34769 (local address 127.16.98.130:45473)
0807 22:49:36.350345 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:36.350347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:36.350389 (+ 42us) server_negotiation.cc:407] Connection header received
0807 22:49:36.350417 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:36.350418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:36.350428 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:36.350475 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:36.350680 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.351216 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.352250 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.352694 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.352789 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.352800 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:36.352949 (+ 149us) server_negotiation.cc:299] Negotiation successful
0807 22:49:36.352990 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 22:49:36.353080 2256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.350163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:45473 (local address 127.16.98.131:34769)
0807 22:49:36.350241 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:36.350256 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:36.350347 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:36.350503 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:36.350504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:36.350506 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:36.350586 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.350588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.351314 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.351315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.352131 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.352134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.352786 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.352787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.352871 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.352874 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:49:36.352944 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 22:49:36.352986 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 22:49:36.353538 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.350169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:40437 (local address 127.16.98.131:53619)
0807 22:49:36.350702 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:36.350717 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:36.350799 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:36.350958 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:36.350959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:36.350961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:36.351042 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.351044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.351737 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.351738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.352634 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.352637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.353267 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.353268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.353346 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.353351 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:49:36.353413 (+ 62us) client_negotiation.cc:240] Negotiation successful
0807 22:49:36.353450 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":343,"thread_start_us":121,"threads_started":1}
I20260807 22:49:36.353470 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f601f2b2428a42898407b304fb105cd1" candidate_uuid: "990e083672fb44a1b05a5e7747bea695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "760ca0f4bdea40899dbac334f6ee2324" is_pre_election: true
I20260807 22:49:36.353572 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.350286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:53619 (local address 127.16.98.129:40437)
0807 22:49:36.350355 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:36.350358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:36.350798 (+ 440us) server_negotiation.cc:407] Connection header received
0807 22:49:36.350856 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:36.350857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:36.350873 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:36.350953 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:36.351124 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.351643 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.352736 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.353168 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.353252 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.353262 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:36.353433 (+ 171us) server_negotiation.cc:299] Negotiation successful
0807 22:49:36.353486 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 22:49:36.353667 2069 raft_consensus.cc:2454] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 990e083672fb44a1b05a5e7747bea695 in term 0.
I20260807 22:49:36.353892 1932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f601f2b2428a42898407b304fb105cd1" candidate_uuid: "990e083672fb44a1b05a5e7747bea695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a92ae26c0bd42a880fb07b575a13f3f" is_pre_election: true
I20260807 22:49:36.353941 2140 leader_election.cc:302] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 760ca0f4bdea40899dbac334f6ee2324, 990e083672fb44a1b05a5e7747bea695; no voters:
I20260807 22:49:36.354036 1932 raft_consensus.cc:2454] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 990e083672fb44a1b05a5e7747bea695 in term 0.
I20260807 22:49:36.354090 2278 raft_consensus.cc:2781] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:49:36.354166 2278 raft_consensus.cc:482] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:49:36.354207 2278 raft_consensus.cc:3037] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:36.355070 2278 raft_consensus.cc:504] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760ca0f4bdea40899dbac334f6ee2324" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45473 } } peers { permanent_uuid: "5a92ae26c0bd42a880fb07b575a13f3f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40437 } } peers { permanent_uuid: "990e083672fb44a1b05a5e7747bea695" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 33673 } }
I20260807 22:49:36.355242 2278 leader_election.cc:288] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [CANDIDATE]: Term 1 election: Requested vote from peers 760ca0f4bdea40899dbac334f6ee2324 (127.16.98.130:45473), 5a92ae26c0bd42a880fb07b575a13f3f (127.16.98.129:40437)
I20260807 22:49:36.355389 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f601f2b2428a42898407b304fb105cd1" candidate_uuid: "990e083672fb44a1b05a5e7747bea695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "760ca0f4bdea40899dbac334f6ee2324"
I20260807 22:49:36.355451 1932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f601f2b2428a42898407b304fb105cd1" candidate_uuid: "990e083672fb44a1b05a5e7747bea695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a92ae26c0bd42a880fb07b575a13f3f"
I20260807 22:49:36.355495 2069 raft_consensus.cc:3037] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:36.355536 1932 raft_consensus.cc:3037] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:36.356487 1932 raft_consensus.cc:2454] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 990e083672fb44a1b05a5e7747bea695 in term 1.
I20260807 22:49:36.356495 2069 raft_consensus.cc:2454] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 990e083672fb44a1b05a5e7747bea695 in term 1.
I20260807 22:49:36.356684 2140 leader_election.cc:302] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 760ca0f4bdea40899dbac334f6ee2324, 990e083672fb44a1b05a5e7747bea695; no voters:
I20260807 22:49:36.356804 2278 raft_consensus.cc:2781] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:36.357019 2278 raft_consensus.cc:686] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [term 1 LEADER]: Becoming Leader. State: Replica: 990e083672fb44a1b05a5e7747bea695, State: Running, Role: LEADER
I20260807 22:49:36.357105 2278 consensus_queue.cc:237] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760ca0f4bdea40899dbac334f6ee2324" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45473 } } peers { permanent_uuid: "5a92ae26c0bd42a880fb07b575a13f3f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40437 } } peers { permanent_uuid: "990e083672fb44a1b05a5e7747bea695" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 33673 } }
I20260807 22:49:36.358142 1781 catalog_manager.cc:5495] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 reported cstate change: term changed from 0 to 1, leader changed from <none> to 990e083672fb44a1b05a5e7747bea695 (127.16.98.131). New cstate: current_term: 1 leader_uuid: "990e083672fb44a1b05a5e7747bea695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760ca0f4bdea40899dbac334f6ee2324" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a92ae26c0bd42a880fb07b575a13f3f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "990e083672fb44a1b05a5e7747bea695" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 33673 } health_report { overall_health: HEALTHY } } }
I20260807 22:49:36.399092 2271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.396393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.16.98.131:33673)
0807 22:49:36.396469 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:36.396471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:36.396481 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:49:36.396532 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:36.396533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:36.396552 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:36.396649 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:49:36.396789 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.397327 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.398235 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.398421 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.398476 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.398616 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:49:36.398840 (+ 224us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:49:36.398912 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:36.398981 (+ 69us) server_negotiation.cc:299] Negotiation successful
0807 22:49:36.399007 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 22:49:36.401335 2069 raft_consensus.cc:1261] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324 [term 1 FOLLOWER]: Refusing update from remote peer 990e083672fb44a1b05a5e7747bea695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:49:36.401378 1932 raft_consensus.cc:1261] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f [term 1 FOLLOWER]: Refusing update from remote peer 990e083672fb44a1b05a5e7747bea695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:49:36.401734 2278 consensus_queue.cc:1035] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "760ca0f4bdea40899dbac334f6ee2324" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:49:36.401935 2282 consensus_queue.cc:1035] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a92ae26c0bd42a880fb07b575a13f3f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 40437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:49:36.404409 2286 mvcc.cc:205] Tried to move back new op lower bound from 7316041631336132608 to 7316041631159500800. Current Snapshot: MvccSnapshot[applied={T|T < 7316041631336132608}]
I20260807 22:49:36.404695 2288 mvcc.cc:205] Tried to move back new op lower bound from 7316041631336132608 to 7316041631159500800. Current Snapshot: MvccSnapshot[applied={T|T < 7316041631336132608}]
I20260807 22:49:36.407341 1780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:49:36.407464 1780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:49:36.408428 1780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c9ed11e4ae2413fa0a33b3d206e5332: Sending DeleteTablet for 3 replicas of tablet f601f2b2428a42898407b304fb105cd1
I20260807 22:49:36.408847 2048 tablet_service.cc:1514] Processing DeleteTablet for tablet f601f2b2428a42898407b304fb105cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:36 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:56502
I20260807 22:49:36.408903 1911 tablet_service.cc:1514] Processing DeleteTablet for tablet f601f2b2428a42898407b304fb105cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:36 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:53404
I20260807 22:49:36.409273 2296 tablet_replica.cc:330] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f: stopping tablet replica
I20260807 22:49:36.409332 2186 tablet_service.cc:1514] Processing DeleteTablet for tablet f601f2b2428a42898407b304fb105cd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:36 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:56740
I20260807 22:49:36.409399 2296 raft_consensus.cc:2229] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:49:36.409454 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 1847
I20260807 22:49:36.409549 2296 raft_consensus.cc:2258] T f601f2b2428a42898407b304fb105cd1 P 5a92ae26c0bd42a880fb07b575a13f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:49:36.409727 2297 tablet_replica.cc:330] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695: stopping tablet replica
I20260807 22:49:36.409870 2297 raft_consensus.cc:2229] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:49:36.410030 2297 raft_consensus.cc:2258] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:49:36.410544 2297 ts_tablet_manager.cc:1905] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:49:36.412633 2297 ts_tablet_manager.cc:1918] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:49:36.412791 2297 log.cc:1196] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/f601f2b2428a42898407b304fb105cd1
I20260807 22:49:36.413383 2297 ts_tablet_manager.cc:1939] T f601f2b2428a42898407b304fb105cd1 P 990e083672fb44a1b05a5e7747bea695: Deleting consensus metadata
I20260807 22:49:36.414017 1767 catalog_manager.cc:4841] TS 990e083672fb44a1b05a5e7747bea695 (127.16.98.131:33673): tablet f601f2b2428a42898407b304fb105cd1 (table test-table [id=bd78dd3a22e04388a09bce53ae213daa]) successfully deleted
W20260807 22:49:36.417989 1767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:40437 (error 108)
I20260807 22:49:36.418105 2294 tablet_replica.cc:330] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324: stopping tablet replica
I20260807 22:49:36.418251 2294 raft_consensus.cc:2229] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:49:36.418387 2294 raft_consensus.cc:2258] T f601f2b2428a42898407b304fb105cd1 P 760ca0f4bdea40899dbac334f6ee2324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:49:36.418718 2267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.418563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:40437 (local address 127.0.0.1:53408)
0807 22:49:36.418627 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:36.418671 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260807 22:49:36.418840 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 1983
W20260807 22:49:36.419865 1767 catalog_manager.cc:4584] TS 5a92ae26c0bd42a880fb07b575a13f3f (127.16.98.129:40437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f601f2b2428a42898407b304fb105cd1: Network error: Client connection negotiation failed: client connection to 127.16.98.129:40437: connect: Connection refused (error 111)
I20260807 22:49:36.426667 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 2120
I20260807 22:49:36.426789 2267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.426604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:45473 (local address 127.0.0.1:56504)
0807 22:49:36.426689 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:36.426728 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.130:45473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 22:49:36.432715 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 1749
2026-08-07T22:49:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2329 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-07T22:49:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:36Z Disabled control of system clock
I20260807 22:49:36.465000 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:37909
--webserver_interface=127.16.98.190
--webserver_port=0
--builtin_ntp_servers=127.16.98.148:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:37909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 22:49:36.550704 2302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:36.550941 2302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:36.550963 2302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:36.552170 2302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:36.552215 2302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:36.552228 2302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:36.552241 2302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:36.552254 2302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:36.553576 2302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:37909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:37909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:36.553791 2302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:36.554054 2302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:36.559381 2307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:36.559414 2308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:36.559471 2310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:36.559648 2302 server_base.cc:1034] running on GCE node
I20260807 22:49:36.560034 2302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:36.560317 2302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:36.561506 2302 hybrid_clock.cc:648] HybridClock initialized: now 1786142976561476 us; error 54 us; skew 500 ppm
I20260807 22:49:36.563078 2302 webserver.cc:460] Webserver started at http://127.16.98.190:35505/ using document root <none> and password file <none>
I20260807 22:49:36.563350 2302 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:36.563401 2302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:36.563568 2302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:36.564739 2302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "deac905d04804b68b7c8b505b85faf20"
format_stamp: "Formatted at 2026-08-07 22:49:36 on dist-test-slave-rgb1"
server_key: "8458cb59fe978efbf3c2438a0e0e9a21"
server_key_iv: "f2a5eb6b10d98844e645bef1a8d66b99"
server_key_version: "encryptionkey@0"
I20260807 22:49:36.565127 2302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "deac905d04804b68b7c8b505b85faf20"
format_stamp: "Formatted at 2026-08-07 22:49:36 on dist-test-slave-rgb1"
server_key: "8458cb59fe978efbf3c2438a0e0e9a21"
server_key_iv: "f2a5eb6b10d98844e645bef1a8d66b99"
server_key_version: "encryptionkey@0"
I20260807 22:49:36.566697 2302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 22:49:36.567533 2317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:36.567735 2302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260807 22:49:36.567836 2302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "deac905d04804b68b7c8b505b85faf20"
format_stamp: "Formatted at 2026-08-07 22:49:36 on dist-test-slave-rgb1"
server_key: "8458cb59fe978efbf3c2438a0e0e9a21"
server_key_iv: "f2a5eb6b10d98844e645bef1a8d66b99"
server_key_version: "encryptionkey@0"
I20260807 22:49:36.567925 2302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:36.582624 2302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:36.582990 2302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:36.583159 2302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:36.587541 2302 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:37909
I20260807 22:49:36.587598 2372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:37909 every 8 connection(s)
I20260807 22:49:36.587996 2302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:36.589154 2373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:36.590058 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 2302
I20260807 22:49:36.590183 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:36.590406 16778 external_mini_cluster.cc:1428] Setting key ae72e173d4bda4d1d9e869a02424b00b
I20260807 22:49:36.592696 2373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20: Bootstrap starting.
I20260807 22:49:36.593366 2373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:36.593691 2373 log.cc:824] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:36.594449 2373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20: No bootstrap required, opened a new log
I20260807 22:49:36.596484 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.591095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.16.98.190:37909)
0807 22:49:36.591295 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:36.591303 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:36.591329 (+ 26us) server_negotiation.cc:407] Connection header received
0807 22:49:36.591755 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:36.591761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:36.591821 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:36.591951 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:36.592219 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.593267 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.594314 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.594778 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.594869 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.595709 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:36.595713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:36.595714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:36.595913 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:36.595952 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:36.595964 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:36.596028 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:36.596147 (+ 119us) server_negotiation.cc:299] Negotiation successful
0807 22:49:36.596251 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260807 22:49:36.596832 2373 raft_consensus.cc:348] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deac905d04804b68b7c8b505b85faf20" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37909 } }
I20260807 22:49:36.596985 2373 raft_consensus.cc:374] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:36.597010 2373 raft_consensus.cc:729] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deac905d04804b68b7c8b505b85faf20, State: Initialized, Role: FOLLOWER
I20260807 22:49:36.597137 2373 consensus_queue.cc:260] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deac905d04804b68b7c8b505b85faf20" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37909 } }
I20260807 22:49:36.597215 2373 raft_consensus.cc:388] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:36.597259 2373 raft_consensus.cc:482] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:36.597316 2373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:36.597877 2373 raft_consensus.cc:504] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deac905d04804b68b7c8b505b85faf20" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37909 } }
I20260807 22:49:36.597965 2373 leader_election.cc:302] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: deac905d04804b68b7c8b505b85faf20; no voters:
I20260807 22:49:36.598106 2373 leader_election.cc:288] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:36.598187 2379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:36.598331 2379 raft_consensus.cc:686] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [term 1 LEADER]: Becoming Leader. State: Replica: deac905d04804b68b7c8b505b85faf20, State: Running, Role: LEADER
I20260807 22:49:36.598366 2373 sys_catalog.cc:564] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:36.598445 2379 consensus_queue.cc:237] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deac905d04804b68b7c8b505b85faf20" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37909 } }
I20260807 22:49:36.598809 2380 sys_catalog.cc:455] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "deac905d04804b68b7c8b505b85faf20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deac905d04804b68b7c8b505b85faf20" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37909 } } }
I20260807 22:49:36.598927 2380 sys_catalog.cc:458] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:36.598811 2381 sys_catalog.cc:455] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader deac905d04804b68b7c8b505b85faf20. Latest consensus state: current_term: 1 leader_uuid: "deac905d04804b68b7c8b505b85faf20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deac905d04804b68b7c8b505b85faf20" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37909 } } }
I20260807 22:49:36.599066 2381 sys_catalog.cc:458] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:36.599207 2387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:36.599696 2387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:36.601284 2387 catalog_manager.cc:1347] Generated new cluster ID: d7e35076eb22423da87259979378044e
I20260807 22:49:36.601346 2387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:49:36.639325 2387 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:36.640148 2387 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:36.656857 2387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20: Generated new TSK 0
I20260807 22:49:36.657156 2387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:49:36.671836 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37909
--builtin_ntp_servers=127.16.98.148:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 22:49:36.769733 2399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:36.769944 2399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:36.769968 2399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:36.771126 2399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:36.771167 2399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:36.771194 2399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:49:36.772629 2399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:36.772831 2399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:36.773088 2399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:36.778380 2405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:36.778389 2407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:36.778489 2404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:36.778635 2399 server_base.cc:1034] running on GCE node
I20260807 22:49:36.778884 2399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:36.779165 2399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:36.780368 2399 hybrid_clock.cc:648] HybridClock initialized: now 1786142976780366 us; error 47 us; skew 500 ppm
I20260807 22:49:36.781905 2399 webserver.cc:460] Webserver started at http://127.16.98.129:41311/ using document root <none> and password file <none>
I20260807 22:49:36.782223 2399 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:36.782306 2399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:36.782480 2399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:36.783803 2399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "d04d288770e04dff98d6f702341f4d80"
format_stamp: "Formatted at 2026-08-07 22:49:36 on dist-test-slave-rgb1"
server_key: "0a69cf5fb245c3601e2068c6b915d15f"
server_key_iv: "08e102386782efad90eb3c8469893c0b"
server_key_version: "encryptionkey@0"
I20260807 22:49:36.784209 2399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "d04d288770e04dff98d6f702341f4d80"
format_stamp: "Formatted at 2026-08-07 22:49:36 on dist-test-slave-rgb1"
server_key: "0a69cf5fb245c3601e2068c6b915d15f"
server_key_iv: "08e102386782efad90eb3c8469893c0b"
server_key_version: "encryptionkey@0"
I20260807 22:49:36.785825 2399 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:49:36.786705 2413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:36.786887 2399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:36.786976 2399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "d04d288770e04dff98d6f702341f4d80"
format_stamp: "Formatted at 2026-08-07 22:49:36 on dist-test-slave-rgb1"
server_key: "0a69cf5fb245c3601e2068c6b915d15f"
server_key_iv: "08e102386782efad90eb3c8469893c0b"
server_key_version: "encryptionkey@0"
I20260807 22:49:36.787035 2399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:36.796483 2399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:36.796838 2399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:36.796953 2399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:36.797140 2399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:36.797489 2399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:36.797524 2399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:36.797554 2399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:36.797565 2399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:36.803819 2399 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:44517
I20260807 22:49:36.803992 2532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:44517 every 8 connection(s)
I20260807 22:49:36.804265 2399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:49:36.808959 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 2399
I20260807 22:49:36.809110 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:49:36.809378 16778 external_mini_cluster.cc:1428] Setting key 2043e575986fe94a340a42ec933ffb75
I20260807 22:49:36.810547 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.805104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:40545 (local address 127.16.98.190:37909)
0807 22:49:36.805198 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:36.805200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:36.805482 (+ 282us) server_negotiation.cc:407] Connection header received
0807 22:49:36.806056 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:36.806057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:36.806074 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:36.806144 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:36.806875 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.807488 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.808513 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.808940 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.809059 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.810158 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:36.810159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:36.810159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:36.810165 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:36.810190 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:36.810195 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:36.810254 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:36.810381 (+ 127us) server_negotiation.cc:299] Negotiation successful
0807 22:49:36.810414 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 22:49:36.810676 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37909
--builtin_ntp_servers=127.16.98.148:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 22:49:36.810784 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.805148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:37909 (local address 127.16.98.129:40545)
0807 22:49:36.805357 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:36.805392 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:36.805927 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:36.806210 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:36.806213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:36.806222 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:36.806784 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.806789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.807614 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.807615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.808424 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.808427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.809086 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.809087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.809298 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.810010 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:49:36.810010 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:36.810041 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:49:36.810067 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:36.810273 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:36.810281 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:36.810285 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:49:36.810363 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 22:49:36.810497 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260807 22:49:36.811496 2533 heartbeater.cc:347] Connected to a master server at 127.16.98.190:37909
I20260807 22:49:36.811646 2533 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:36.811952 2533 heartbeater.cc:511] Master 127.16.98.190:37909 requested a full tablet report, sending...
I20260807 22:49:36.812634 2334 ts_manager.cc:194] Registered new tserver with Master: d04d288770e04dff98d6f702341f4d80 (127.16.98.129:44517)
I20260807 22:49:36.813319 2334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.98.129:40545
W20260807 22:49:36.904281 2536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:36.904537 2536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:36.904623 2536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:36.905956 2536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:36.906005 2536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:36.906035 2536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:49:36.907563 2536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:36.907843 2536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:36.908196 2536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:36.914440 2541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:36.914479 2542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:36.914440 2544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:36.914839 2536 server_base.cc:1034] running on GCE node
I20260807 22:49:36.915103 2536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:36.915344 2536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:36.916538 2536 hybrid_clock.cc:648] HybridClock initialized: now 1786142976916530 us; error 56 us; skew 500 ppm
I20260807 22:49:36.918035 2536 webserver.cc:460] Webserver started at http://127.16.98.130:37143/ using document root <none> and password file <none>
I20260807 22:49:36.918298 2536 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:36.918368 2536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:36.918526 2536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:36.919689 2536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb"
format_stamp: "Formatted at 2026-08-07 22:49:36 on dist-test-slave-rgb1"
server_key: "c7ebe3003ec3920d56dd28096c660909"
server_key_iv: "11b9bc1e4eaf67a89e30b7d41639f282"
server_key_version: "encryptionkey@0"
I20260807 22:49:36.920076 2536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb"
format_stamp: "Formatted at 2026-08-07 22:49:36 on dist-test-slave-rgb1"
server_key: "c7ebe3003ec3920d56dd28096c660909"
server_key_iv: "11b9bc1e4eaf67a89e30b7d41639f282"
server_key_version: "encryptionkey@0"
I20260807 22:49:36.921687 2536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260807 22:49:36.922636 2550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:36.922976 2536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:36.923131 2536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb"
format_stamp: "Formatted at 2026-08-07 22:49:36 on dist-test-slave-rgb1"
server_key: "c7ebe3003ec3920d56dd28096c660909"
server_key_iv: "11b9bc1e4eaf67a89e30b7d41639f282"
server_key_version: "encryptionkey@0"
I20260807 22:49:36.923218 2536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:36.940222 2536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:36.940665 2536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:36.940846 2536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:36.941145 2536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:36.941532 2536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:36.941593 2536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:36.941639 2536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:36.941669 2536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:36.948117 2536 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:39775
I20260807 22:49:36.948165 2666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:39775 every 8 connection(s)
I20260807 22:49:36.948521 2536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:49:36.954844 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.949479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:51323 (local address 127.16.98.190:37909)
0807 22:49:36.949587 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:36.949589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:36.949793 (+ 204us) server_negotiation.cc:407] Connection header received
0807 22:49:36.950320 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:36.950322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:36.950333 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:36.950407 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:36.951147 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.951747 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.952902 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.953122 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.953205 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.954414 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:36.954416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:36.954416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:36.954421 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:36.954437 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:36.954442 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:36.954501 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:36.954697 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 22:49:36.954762 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 22:49:36.955006 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:36.949462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:37909 (local address 127.16.98.130:51323)
0807 22:49:36.949657 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:36.949700 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:36.950208 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:36.950484 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:36.950487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:36.950496 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:36.951031 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.951036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.951844 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.951845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.952767 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:36.952770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:36.953247 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:36.953249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:36.953383 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:36.954231 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:49:36.954232 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:36.954261 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:49:36.954286 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:36.954537 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:36.954544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:36.954547 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:49:36.954626 (+ 79us) client_negotiation.cc:240] Negotiation successful
0807 22:49:36.954764 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260807 22:49:36.955705 2667 heartbeater.cc:347] Connected to a master server at 127.16.98.190:37909
I20260807 22:49:36.955802 2667 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:36.956087 2667 heartbeater.cc:511] Master 127.16.98.190:37909 requested a full tablet report, sending...
I20260807 22:49:36.956542 2334 ts_manager.cc:194] Registered new tserver with Master: 4fb09d9b6c6e4f138e2b3d0757164fdb (127.16.98.130:39775)
I20260807 22:49:36.957113 2334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.98.130:51323
I20260807 22:49:36.957824 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 2536
I20260807 22:49:36.957947 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:49:36.958166 16778 external_mini_cluster.cc:1428] Setting key edc1c92a14e9b8277cf70223464c2323
I20260807 22:49:36.959231 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37909
--builtin_ntp_servers=127.16.98.148:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 22:49:37.050562 2670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:37.050789 2670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:37.050813 2670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:37.051905 2670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:37.051941 2670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:37.051966 2670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:49:37.053297 2670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:37.053501 2670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:37.053755 2670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 22:49:37.059296 2670 server_base.cc:1034] running on GCE node
W20260807 22:49:37.059235 2675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:37.059217 2676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:37.059267 2678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:37.059749 2670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:37.060043 2670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:37.061233 2670 hybrid_clock.cc:648] HybridClock initialized: now 1786142977061227 us; error 38 us; skew 500 ppm
I20260807 22:49:37.062670 2670 webserver.cc:460] Webserver started at http://127.16.98.131:41859/ using document root <none> and password file <none>
I20260807 22:49:37.062955 2670 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:37.063033 2670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:37.063176 2670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:37.064262 2670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "3870bda6aaaa46cc85bd3ab23cfcc856"
format_stamp: "Formatted at 2026-08-07 22:49:37 on dist-test-slave-rgb1"
server_key: "5fb5a81bba8dce842877e9380f350d26"
server_key_iv: "74a44b1d172bd67da617274bd076c83a"
server_key_version: "encryptionkey@0"
I20260807 22:49:37.064706 2670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "3870bda6aaaa46cc85bd3ab23cfcc856"
format_stamp: "Formatted at 2026-08-07 22:49:37 on dist-test-slave-rgb1"
server_key: "5fb5a81bba8dce842877e9380f350d26"
server_key_iv: "74a44b1d172bd67da617274bd076c83a"
server_key_version: "encryptionkey@0"
I20260807 22:49:37.066252 2670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:49:37.067354 2684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:37.067627 2670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:49:37.067744 2670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "3870bda6aaaa46cc85bd3ab23cfcc856"
format_stamp: "Formatted at 2026-08-07 22:49:37 on dist-test-slave-rgb1"
server_key: "5fb5a81bba8dce842877e9380f350d26"
server_key_iv: "74a44b1d172bd67da617274bd076c83a"
server_key_version: "encryptionkey@0"
I20260807 22:49:37.067893 2670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:37.077544 2670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:37.077883 2670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:37.078042 2670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:37.078285 2670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:37.078629 2670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:37.078683 2670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:37.078725 2670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:37.078756 2670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:37.086287 2670 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:45089
I20260807 22:49:37.086326 2798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:45089 every 8 connection(s)
I20260807 22:49:37.086762 2670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:49:37.092517 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:37.087571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:48271 (local address 127.16.98.190:37909)
0807 22:49:37.087648 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:37.087651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:37.087901 (+ 250us) server_negotiation.cc:407] Connection header received
0807 22:49:37.088466 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:37.088468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:37.088479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:37.088533 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:37.089366 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.089868 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.090991 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.091185 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.091243 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:37.092165 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:37.092166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:37.092166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:37.092171 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:37.092184 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:37.092187 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:37.092237 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:37.092433 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 22:49:37.092454 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 22:49:37.092900 2801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:37.087575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:37909 (local address 127.16.98.131:48271)
0807 22:49:37.087771 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:37.087809 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:37.088368 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:37.088666 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:37.088670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:37.088679 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:37.089271 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:37.089279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.089995 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.089996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:37.090902 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:37.090905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.091268 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.091269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:37.091383 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:37.092005 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:49:37.092006 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:37.092046 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:49:37.092070 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:37.092285 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:37.092293 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:37.092296 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:49:37.092427 (+ 131us) client_negotiation.cc:240] Negotiation successful
0807 22:49:37.092529 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260807 22:49:37.093483 2799 heartbeater.cc:347] Connected to a master server at 127.16.98.190:37909
I20260807 22:49:37.093581 2799 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:37.093734 2799 heartbeater.cc:511] Master 127.16.98.190:37909 requested a full tablet report, sending...
I20260807 22:49:37.094139 2334 ts_manager.cc:194] Registered new tserver with Master: 3870bda6aaaa46cc85bd3ab23cfcc856 (127.16.98.131:45089)
I20260807 22:49:37.094508 2334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.98.131:48271
I20260807 22:49:37.095530 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 2670
I20260807 22:49:37.095625 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:49:37.095791 16778 external_mini_cluster.cc:1428] Setting key 759f823190a7e4ae025dc312251f270c
I20260807 22:49:37.096890 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:49:37.102265 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:37.098369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.16.98.190:37909)
0807 22:49:37.098432 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:37.098434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:37.098567 (+ 133us) server_negotiation.cc:407] Connection header received
0807 22:49:37.098613 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:37.098613 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:37.098622 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:37.098675 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:37.099153 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.099767 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.100788 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.101009 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.101092 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:37.101897 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:37.101899 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:37.101899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:37.101905 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:37.101920 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:37.101924 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:37.102058 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:37.102148 (+ 90us) server_negotiation.cc:299] Negotiation successful
0807 22:49:37.102173 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260807 22:49:37.104033 2334 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:49:37.105788 2334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:49:37.113232 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:37.108638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.16.98.129:44517)
0807 22:49:37.108847 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:37.108852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:37.108918 (+ 66us) server_negotiation.cc:407] Connection header received
0807 22:49:37.109008 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:37.109010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:37.109043 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:37.109122 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:37.109326 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.110451 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.111904 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.112738 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.112827 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:37.112934 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:37.113072 (+ 138us) server_negotiation.cc:299] Negotiation successful
0807 22:49:37.113119 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
I20260807 22:49:37.113284 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:37.108323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:44517 (local address 127.0.0.1:58016)
0807 22:49:37.108839 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:37.108853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:37.108932 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:37.109193 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:37.109194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:37.109196 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:37.109243 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:37.109245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.110738 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.110739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:37.111803 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:37.111807 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.112837 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.112838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:37.112931 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:37.112950 (+ 19us) client_negotiation.cc:769] Sending connection context
0807 22:49:37.113046 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 22:49:37.113132 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":81,"spinlock_wait_cycles":768,"thread_start_us":84,"threads_started":1}
I20260807 22:49:37.113268 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:37.108321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:45089 (local address 127.0.0.1:47286)
0807 22:49:37.108497 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:37.108551 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:37.108663 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:37.109072 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:37.109075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:37.109078 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:37.109158 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:37.109160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.110558 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.110559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:37.111933 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:37.111935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.112858 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.112859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:37.112945 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:37.112950 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:49:37.113051 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 22:49:37.113132 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":138,"thread_start_us":62,"threads_started":1}
I20260807 22:49:37.114120 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:37.108323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:39775 (local address 127.0.0.1:55550)
0807 22:49:37.109041 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:37.109051 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:37.109200 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:37.109451 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:37.109451 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:37.109458 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:37.109505 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:37.109507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.110599 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.110600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:37.111932 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:37.111935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.113775 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.113776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:37.113841 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:37.113845 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:49:37.113992 (+ 147us) client_negotiation.cc:240] Negotiation successful
0807 22:49:37.114041 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":6,"thread_start_us":64,"threads_started":1}
I20260807 22:49:37.114166 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:37.108619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55550 (local address 127.16.98.130:39775)
0807 22:49:37.108847 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:37.108853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:37.109169 (+ 316us) server_negotiation.cc:407] Connection header received
0807 22:49:37.109315 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:37.109318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:37.109334 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:37.109553 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:37.109604 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.110514 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.112674 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.113560 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.113639 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:37.113734 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:37.114034 (+ 300us) server_negotiation.cc:299] Negotiation successful
0807 22:49:37.114080 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":113,"threads_started":1}
I20260807 22:49:37.114198 2463 tablet_service.cc:1467] Processing CreateTablet for tablet a600058a17744ad5801a0d2c83d6e0bf (DEFAULT_TABLE table=test-table [id=4ebc566e2b954a8c859333e92c22c353]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:37.114166 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:37.108426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47286 (local address 127.16.98.131:45089)
0807 22:49:37.108880 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:37.108885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:37.108896 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:49:37.108940 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:37.108942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:37.108959 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:37.109050 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:37.109256 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.110451 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.112039 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.112759 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.113880 (+ 1121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:37.113997 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:37.114026 (+ 29us) server_negotiation.cc:299] Negotiation successful
0807 22:49:37.114080 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":92,"threads_started":1}
I20260807 22:49:37.114711 2463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a600058a17744ad5801a0d2c83d6e0bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:37.114846 2599 tablet_service.cc:1467] Processing CreateTablet for tablet a600058a17744ad5801a0d2c83d6e0bf (DEFAULT_TABLE table=test-table [id=4ebc566e2b954a8c859333e92c22c353]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:37.115199 2599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a600058a17744ad5801a0d2c83d6e0bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:37.117930 2813 tablet_bootstrap.cc:492] T a600058a17744ad5801a0d2c83d6e0bf P d04d288770e04dff98d6f702341f4d80: Bootstrap starting.
I20260807 22:49:37.117947 2814 tablet_bootstrap.cc:492] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb: Bootstrap starting.
I20260807 22:49:37.118511 2732 tablet_service.cc:1467] Processing CreateTablet for tablet a600058a17744ad5801a0d2c83d6e0bf (DEFAULT_TABLE table=test-table [id=4ebc566e2b954a8c859333e92c22c353]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:37.118677 2814 tablet_bootstrap.cc:654] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:37.118757 2813 tablet_bootstrap.cc:654] T a600058a17744ad5801a0d2c83d6e0bf P d04d288770e04dff98d6f702341f4d80: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:37.118908 2732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a600058a17744ad5801a0d2c83d6e0bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:37.119041 2814 log.cc:824] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:37.119140 2813 log.cc:824] T a600058a17744ad5801a0d2c83d6e0bf P d04d288770e04dff98d6f702341f4d80: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:37.119859 2814 tablet_bootstrap.cc:492] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb: No bootstrap required, opened a new log
I20260807 22:49:37.119930 2814 ts_tablet_manager.cc:1397] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:49:37.120054 2813 tablet_bootstrap.cc:492] T a600058a17744ad5801a0d2c83d6e0bf P d04d288770e04dff98d6f702341f4d80: No bootstrap required, opened a new log
I20260807 22:49:37.120126 2813 ts_tablet_manager.cc:1397] T a600058a17744ad5801a0d2c83d6e0bf P d04d288770e04dff98d6f702341f4d80: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:49:37.121980 2814 raft_consensus.cc:348] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d04d288770e04dff98d6f702341f4d80" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44517 } } peers { permanent_uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 39775 } } peers { permanent_uuid: "3870bda6aaaa46cc85bd3ab23cfcc856" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 45089 } }
I20260807 22:49:37.122151 2814 raft_consensus.cc:374] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:37.122200 2814 raft_consensus.cc:729] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fb09d9b6c6e4f138e2b3d0757164fdb, State: Initialized, Role: FOLLOWER
I20260807 22:49:37.122331 2814 consensus_queue.cc:260] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d04d288770e04dff98d6f702341f4d80" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44517 } } peers { permanent_uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 39775 } } peers { permanent_uuid: "3870bda6aaaa46cc85bd3ab23cfcc856" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 45089 } }
I20260807 22:49:37.122684 2667 heartbeater.cc:503] Master 127.16.98.190:37909 was elected leader, sending a full tablet report...
I20260807 22:49:37.123041 2818 tablet_bootstrap.cc:492] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856: Bootstrap starting.
I20260807 22:49:37.123292 2814 ts_tablet_manager.cc:1428] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 22:49:37.123245 2813 raft_consensus.cc:348] T a600058a17744ad5801a0d2c83d6e0bf P d04d288770e04dff98d6f702341f4d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d04d288770e04dff98d6f702341f4d80" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44517 } } peers { permanent_uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 39775 } } peers { permanent_uuid: "3870bda6aaaa46cc85bd3ab23cfcc856" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 45089 } }
I20260807 22:49:37.123431 2813 raft_consensus.cc:374] T a600058a17744ad5801a0d2c83d6e0bf P d04d288770e04dff98d6f702341f4d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:37.123483 2813 raft_consensus.cc:729] T a600058a17744ad5801a0d2c83d6e0bf P d04d288770e04dff98d6f702341f4d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04d288770e04dff98d6f702341f4d80, State: Initialized, Role: FOLLOWER
I20260807 22:49:37.123606 2813 consensus_queue.cc:260] T a600058a17744ad5801a0d2c83d6e0bf P d04d288770e04dff98d6f702341f4d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d04d288770e04dff98d6f702341f4d80" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44517 } } peers { permanent_uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 39775 } } peers { permanent_uuid: "3870bda6aaaa46cc85bd3ab23cfcc856" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 45089 } }
I20260807 22:49:37.123849 2818 tablet_bootstrap.cc:654] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:37.123915 2813 ts_tablet_manager.cc:1428] T a600058a17744ad5801a0d2c83d6e0bf P d04d288770e04dff98d6f702341f4d80: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260807 22:49:37.123992 2533 heartbeater.cc:503] Master 127.16.98.190:37909 was elected leader, sending a full tablet report...
I20260807 22:49:37.124217 2818 log.cc:824] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:37.125240 2818 tablet_bootstrap.cc:492] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856: No bootstrap required, opened a new log
I20260807 22:49:37.125311 2818 ts_tablet_manager.cc:1397] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:49:37.127238 2818 raft_consensus.cc:348] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d04d288770e04dff98d6f702341f4d80" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44517 } } peers { permanent_uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 39775 } } peers { permanent_uuid: "3870bda6aaaa46cc85bd3ab23cfcc856" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 45089 } }
I20260807 22:49:37.127391 2818 raft_consensus.cc:374] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:37.127439 2818 raft_consensus.cc:729] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3870bda6aaaa46cc85bd3ab23cfcc856, State: Initialized, Role: FOLLOWER
I20260807 22:49:37.127562 2818 consensus_queue.cc:260] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d04d288770e04dff98d6f702341f4d80" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44517 } } peers { permanent_uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 39775 } } peers { permanent_uuid: "3870bda6aaaa46cc85bd3ab23cfcc856" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 45089 } }
I20260807 22:49:37.127823 2818 ts_tablet_manager.cc:1428] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 22:49:37.127915 2799 heartbeater.cc:503] Master 127.16.98.190:37909 was elected leader, sending a full tablet report...
W20260807 22:49:37.199375 2668 tablet.cc:2369] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:49:37.265085 2817 raft_consensus.cc:482] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:49:37.265275 2817 raft_consensus.cc:504] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d04d288770e04dff98d6f702341f4d80" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44517 } } peers { permanent_uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 39775 } } peers { permanent_uuid: "3870bda6aaaa46cc85bd3ab23cfcc856" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 45089 } }
I20260807 22:49:37.265751 2817 leader_election.cc:288] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d04d288770e04dff98d6f702341f4d80 (127.16.98.129:44517), 3870bda6aaaa46cc85bd3ab23cfcc856 (127.16.98.131:45089)
I20260807 22:49:37.269251 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:37.266111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:48537 (local address 127.16.98.129:44517)
0807 22:49:37.266160 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:37.266163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:37.266356 (+ 193us) server_negotiation.cc:407] Connection header received
0807 22:49:37.266446 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:37.266447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:37.266460 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:37.266531 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:37.266726 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.267241 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.268353 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.268858 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.268948 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:37.268958 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:37.269118 (+ 160us) server_negotiation.cc:299] Negotiation successful
0807 22:49:37.269174 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 22:49:37.269243 2822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:37.266049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:44517 (local address 127.16.98.130:48537)
0807 22:49:37.266263 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:37.266275 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:37.266352 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:37.266541 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:37.266541 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:37.266543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:37.266643 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:37.266645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.267337 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.267338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:37.268239 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:37.268242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.268943 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.268945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:37.269038 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:37.269042 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:49:37.269090 (+ 48us) client_negotiation.cc:240] Negotiation successful
0807 22:49:37.269124 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":2,"thread_start_us":137,"threads_started":1}
I20260807 22:49:37.269315 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:37.266083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:48997 (local address 127.16.98.131:45089)
0807 22:49:37.266140 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:37.266142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:37.266196 (+ 54us) server_negotiation.cc:407] Connection header received
0807 22:49:37.266307 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:37.266309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:37.266321 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:37.266408 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:37.266629 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.267184 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.268377 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.268862 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.268964 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:37.268973 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:37.269182 (+ 209us) server_negotiation.cc:299] Negotiation successful
0807 22:49:37.269228 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 22:49:37.269349 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:37.266026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:45089 (local address 127.16.98.130:48997)
0807 22:49:37.266125 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:37.266140 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:37.266211 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:37.266422 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:37.266423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:37.266425 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:37.266537 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:37.266540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.267283 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.267285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:37.268245 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:37.268247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.268995 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.268996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:37.269090 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:37.269094 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:49:37.269161 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 22:49:37.269203 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":30}
I20260807 22:49:37.270071 2752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a600058a17744ad5801a0d2c83d6e0bf" candidate_uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3870bda6aaaa46cc85bd3ab23cfcc856" is_pre_election: true
I20260807 22:49:37.270073 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a600058a17744ad5801a0d2c83d6e0bf" candidate_uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d04d288770e04dff98d6f702341f4d80" is_pre_election: true
I20260807 22:49:37.270375 2484 raft_consensus.cc:2454] T a600058a17744ad5801a0d2c83d6e0bf P d04d288770e04dff98d6f702341f4d80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fb09d9b6c6e4f138e2b3d0757164fdb in term 0.
I20260807 22:49:37.270375 2752 raft_consensus.cc:2454] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fb09d9b6c6e4f138e2b3d0757164fdb in term 0.
I20260807 22:49:37.270792 2554 leader_election.cc:302] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3870bda6aaaa46cc85bd3ab23cfcc856, 4fb09d9b6c6e4f138e2b3d0757164fdb; no voters:
I20260807 22:49:37.270962 2817 raft_consensus.cc:2781] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:49:37.271060 2817 raft_consensus.cc:482] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:49:37.271085 2817 raft_consensus.cc:3037] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:37.272046 2817 raft_consensus.cc:504] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d04d288770e04dff98d6f702341f4d80" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44517 } } peers { permanent_uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 39775 } } peers { permanent_uuid: "3870bda6aaaa46cc85bd3ab23cfcc856" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 45089 } }
I20260807 22:49:37.272207 2817 leader_election.cc:288] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [CANDIDATE]: Term 1 election: Requested vote from peers d04d288770e04dff98d6f702341f4d80 (127.16.98.129:44517), 3870bda6aaaa46cc85bd3ab23cfcc856 (127.16.98.131:45089)
I20260807 22:49:37.272404 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a600058a17744ad5801a0d2c83d6e0bf" candidate_uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d04d288770e04dff98d6f702341f4d80"
I20260807 22:49:37.272606 2484 raft_consensus.cc:3037] T a600058a17744ad5801a0d2c83d6e0bf P d04d288770e04dff98d6f702341f4d80 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:37.272490 2752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a600058a17744ad5801a0d2c83d6e0bf" candidate_uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3870bda6aaaa46cc85bd3ab23cfcc856"
I20260807 22:49:37.272855 2752 raft_consensus.cc:3037] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:37.273553 2484 raft_consensus.cc:2454] T a600058a17744ad5801a0d2c83d6e0bf P d04d288770e04dff98d6f702341f4d80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fb09d9b6c6e4f138e2b3d0757164fdb in term 1.
I20260807 22:49:37.273748 2752 raft_consensus.cc:2454] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fb09d9b6c6e4f138e2b3d0757164fdb in term 1.
I20260807 22:49:37.273772 2553 leader_election.cc:302] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fb09d9b6c6e4f138e2b3d0757164fdb, d04d288770e04dff98d6f702341f4d80; no voters:
I20260807 22:49:37.274032 2817 raft_consensus.cc:2781] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:37.274224 2817 raft_consensus.cc:686] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [term 1 LEADER]: Becoming Leader. State: Replica: 4fb09d9b6c6e4f138e2b3d0757164fdb, State: Running, Role: LEADER
I20260807 22:49:37.274339 2817 consensus_queue.cc:237] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d04d288770e04dff98d6f702341f4d80" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44517 } } peers { permanent_uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 39775 } } peers { permanent_uuid: "3870bda6aaaa46cc85bd3ab23cfcc856" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 45089 } }
I20260807 22:49:37.275282 2334 catalog_manager.cc:5495] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fb09d9b6c6e4f138e2b3d0757164fdb (127.16.98.130). New cstate: current_term: 1 leader_uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d04d288770e04dff98d6f702341f4d80" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fb09d9b6c6e4f138e2b3d0757164fdb" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 39775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3870bda6aaaa46cc85bd3ab23cfcc856" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 45089 } health_report { overall_health: UNKNOWN } } }
I20260807 22:49:37.296428 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:37.293683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.16.98.130:39775)
0807 22:49:37.293746 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:37.293748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:37.293759 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:49:37.293816 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:37.293817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:37.293839 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:37.294006 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:49:37.294181 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.294705 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.295539 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:37.295750 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:37.295834 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:37.296042 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:49:37.296151 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:49:37.296257 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:37.296279 (+ 22us) server_negotiation.cc:299] Negotiation successful
0807 22:49:37.296302 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260807 22:49:37.298666 2752 raft_consensus.cc:1261] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856 [term 1 FOLLOWER]: Refusing update from remote peer 4fb09d9b6c6e4f138e2b3d0757164fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:49:37.298678 2484 raft_consensus.cc:1261] T a600058a17744ad5801a0d2c83d6e0bf P d04d288770e04dff98d6f702341f4d80 [term 1 FOLLOWER]: Refusing update from remote peer 4fb09d9b6c6e4f138e2b3d0757164fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:49:37.299077 2823 consensus_queue.cc:1035] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d04d288770e04dff98d6f702341f4d80" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:49:37.299250 2817 consensus_queue.cc:1035] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3870bda6aaaa46cc85bd3ab23cfcc856" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 45089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:49:37.301942 2826 mvcc.cc:205] Tried to move back new op lower bound from 7316041635012050944 to 7316041634916360192. Current Snapshot: MvccSnapshot[applied={T|T < 7316041635012050944}]
I20260807 22:49:37.302143 2828 mvcc.cc:205] Tried to move back new op lower bound from 7316041635012050944 to 7316041634916360192. Current Snapshot: MvccSnapshot[applied={T|T < 7316041635012050944}]
I20260807 22:49:37.302764 2827 mvcc.cc:205] Tried to move back new op lower bound from 7316041635012050944 to 7316041634916360192. Current Snapshot: MvccSnapshot[applied={T|T < 7316041635012050944}]
I20260807 22:49:37.304121 2333 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:49:37.304226 2333 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:49:37.305138 2333 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deac905d04804b68b7c8b505b85faf20: Sending DeleteTablet for 3 replicas of tablet a600058a17744ad5801a0d2c83d6e0bf
I20260807 22:49:37.305545 2732 tablet_service.cc:1514] Processing DeleteTablet for tablet a600058a17744ad5801a0d2c83d6e0bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:37 UTC) from {username='slave'} at 127.0.0.1:47286
I20260807 22:49:37.305591 2463 tablet_service.cc:1514] Processing DeleteTablet for tablet a600058a17744ad5801a0d2c83d6e0bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:37 UTC) from {username='slave'} at 127.0.0.1:58016
I20260807 22:49:37.305624 2599 tablet_service.cc:1514] Processing DeleteTablet for tablet a600058a17744ad5801a0d2c83d6e0bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:37 UTC) from {username='slave'} at 127.0.0.1:55550
I20260807 22:49:37.305847 2837 tablet_replica.cc:330] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856: stopping tablet replica
I20260807 22:49:37.305859 2839 tablet_replica.cc:330] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb: stopping tablet replica
I20260807 22:49:37.305951 2837 raft_consensus.cc:2229] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:49:37.306044 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 2399
I20260807 22:49:37.306082 2837 raft_consensus.cc:2258] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:49:37.306218 2839 raft_consensus.cc:2229] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:49:37.306370 2839 raft_consensus.cc:2258] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:49:37.306985 2837 ts_tablet_manager.cc:1905] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:49:37.307338 2839 ts_tablet_manager.cc:1905] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:49:37.309585 2839 ts_tablet_manager.cc:1918] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:49:37.309618 2837 ts_tablet_manager.cc:1918] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:49:37.309688 2837 log.cc:1196] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/a600058a17744ad5801a0d2c83d6e0bf
I20260807 22:49:37.309795 2839 log.cc:1196] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/a600058a17744ad5801a0d2c83d6e0bf
I20260807 22:49:37.309965 2837 ts_tablet_manager.cc:1939] T a600058a17744ad5801a0d2c83d6e0bf P 3870bda6aaaa46cc85bd3ab23cfcc856: Deleting consensus metadata
I20260807 22:49:37.310304 2839 ts_tablet_manager.cc:1939] T a600058a17744ad5801a0d2c83d6e0bf P 4fb09d9b6c6e4f138e2b3d0757164fdb: Deleting consensus metadata
I20260807 22:49:37.310740 2318 catalog_manager.cc:4841] TS 4fb09d9b6c6e4f138e2b3d0757164fdb (127.16.98.130:39775): tablet a600058a17744ad5801a0d2c83d6e0bf (table test-table [id=4ebc566e2b954a8c859333e92c22c353]) successfully deleted
I20260807 22:49:37.310761 2321 catalog_manager.cc:4841] TS 3870bda6aaaa46cc85bd3ab23cfcc856 (127.16.98.131:45089): tablet a600058a17744ad5801a0d2c83d6e0bf (table test-table [id=4ebc566e2b954a8c859333e92c22c353]) successfully deleted
W20260807 22:49:37.313550 2320 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:44517 (error 108)
I20260807 22:49:37.313951 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 2536
I20260807 22:49:37.315449 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:37.315289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:44517 (local address 127.0.0.1:58030)
0807 22:49:37.315360 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:37.315417 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 22:49:37.315627 2320 catalog_manager.cc:4584] TS d04d288770e04dff98d6f702341f4d80 (127.16.98.129:44517): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a600058a17744ad5801a0d2c83d6e0bf: Network error: Client connection negotiation failed: client connection to 127.16.98.129:44517: connect: Connection refused (error 111)
I20260807 22:49:37.321257 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 2670
I20260807 22:49:37.326774 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 2302
2026-08-07T22:49:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (893 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:49:37 dist-test-slave-rgb1 krb5kdc[2843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:49:37 dist-test-slave-rgb1 krb5kdc[2843](info): set up 2 sockets
Aug 07 22:49:37 dist-test-slave-rgb1 krb5kdc[2843](info): commencing operation
krb5kdc: starting...
W20260807 22:49:38.440303 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:49:38 dist-test-slave-rgb1 krb5kdc[2843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:49:38.592975 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43977
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:43977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:38.680154 2856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:38.680457 2856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:38.680506 2856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:38.681926 2856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:38.682018 2856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:38.682058 2856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:38.682097 2856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:38.682132 2856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:38.683863 2856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:43977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:38.684180 2856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:38.684435 2856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:38.690356 2861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:38.690331 2863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:38.690321 2865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:38.691222 2856 server_base.cc:1034] running on GCE node
I20260807 22:49:38.691602 2856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:38.692020 2856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:38.693238 2856 hybrid_clock.cc:648] HybridClock initialized: now 1786142978693072 us; error 202 us; skew 500 ppm
Aug 07 22:49:38 dist-test-slave-rgb1 krb5kdc[2843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142978, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:38.696723 2856 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:49:38.697312 2856 webserver.cc:460] Webserver started at http://127.16.98.190:45923/ using document root <none> and password file <none>
I20260807 22:49:38.697584 2856 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:38.697659 2856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:38.697831 2856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:38.699066 2856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "96186599841a458197c8139470203ab7"
format_stamp: "Formatted at 2026-08-07 22:49:38 on dist-test-slave-rgb1"
server_key: "baa1aca627870e46aa72ee5cfd781b08"
server_key_iv: "42baba612375470bced4bc5841c66197"
server_key_version: "encryptionkey@0"
I20260807 22:49:38.699606 2856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "96186599841a458197c8139470203ab7"
format_stamp: "Formatted at 2026-08-07 22:49:38 on dist-test-slave-rgb1"
server_key: "baa1aca627870e46aa72ee5cfd781b08"
server_key_iv: "42baba612375470bced4bc5841c66197"
server_key_version: "encryptionkey@0"
I20260807 22:49:38.701297 2856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260807 22:49:38.702315 2872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:38.702668 2856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:38.702889 2856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "96186599841a458197c8139470203ab7"
format_stamp: "Formatted at 2026-08-07 22:49:38 on dist-test-slave-rgb1"
server_key: "baa1aca627870e46aa72ee5cfd781b08"
server_key_iv: "42baba612375470bced4bc5841c66197"
server_key_version: "encryptionkey@0"
I20260807 22:49:38.703022 2856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:38.727785 2856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:38.730640 2856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:38.730871 2856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:38.736167 2856 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:43977
I20260807 22:49:38.736258 2924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:43977 every 8 connection(s)
I20260807 22:49:38.736714 2856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:38.737727 2925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:38.739614 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 2856
I20260807 22:49:38.739760 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:38.739969 16778 external_mini_cluster.cc:1428] Setting key 908b868c0dad246c8058c476d7523122
I20260807 22:49:38.740692 2925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7: Bootstrap starting.
I20260807 22:49:38.741325 2925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:38.741647 2925 log.cc:824] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:38.742585 2925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7: No bootstrap required, opened a new log
I20260807 22:49:38.744613 2925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96186599841a458197c8139470203ab7" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43977 } }
I20260807 22:49:38.744824 2925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:38.744879 2925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96186599841a458197c8139470203ab7, State: Initialized, Role: FOLLOWER
I20260807 22:49:38.745038 2925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96186599841a458197c8139470203ab7" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43977 } }
I20260807 22:49:38.745123 2925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:38.745172 2925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:38.745225 2925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:38.745894 2925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96186599841a458197c8139470203ab7" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43977 } }
I20260807 22:49:38.746028 2925 leader_election.cc:302] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96186599841a458197c8139470203ab7; no voters:
I20260807 22:49:38.746253 2925 leader_election.cc:288] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:38.746316 2930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:38.746469 2930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [term 1 LEADER]: Becoming Leader. State: Replica: 96186599841a458197c8139470203ab7, State: Running, Role: LEADER
I20260807 22:49:38.746559 2925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:38.746657 2930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96186599841a458197c8139470203ab7" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43977 } }
I20260807 22:49:38.747219 2932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96186599841a458197c8139470203ab7. Latest consensus state: current_term: 1 leader_uuid: "96186599841a458197c8139470203ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96186599841a458197c8139470203ab7" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43977 } } }
I20260807 22:49:38.747313 2932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:38.747696 2931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96186599841a458197c8139470203ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96186599841a458197c8139470203ab7" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43977 } } }
I20260807 22:49:38.747773 2931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7 [sys.catalog]: This master's current role is: LEADER
Aug 07 22:49:38 dist-test-slave-rgb1 krb5kdc[2843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
W20260807 22:49:38.748633 2947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:49:38.748733 2947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:49:38.748812 2940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:38.749073 2940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:38.750733 2940 catalog_manager.cc:1347] Generated new cluster ID: 48a67fe5d9134fb2adc61155adeff858
I20260807 22:49:38.750788 2940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:49:38.752894 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:38.740609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39738 (local address 127.16.98.190:43977)
0807 22:49:38.740823 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:38.740833 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:38.740880 (+ 47us) server_negotiation.cc:407] Connection header received
0807 22:49:38.741400 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:38.741410 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:38.741490 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:38.741660 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:38.742548 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:38.743616 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:38.744472 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:38.745393 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:38.745541 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:38.748935 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:38.748940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:38.748941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:38.748956 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:38.751123 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:38.751579 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:38.751583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:38.751584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:38.751688 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:38.751922 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:38.751923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:38.751924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:38.752183 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:38.752384 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:38.752480 (+ 96us) server_negotiation.cc:299] Negotiation successful
0807 22:49:38.752666 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":92,"threads_started":1}
I20260807 22:49:38.768056 2940 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:38.768857 2940 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:38.774549 2940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7: Generated new TSK 0
I20260807 22:49:38.774770 2940 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:49:38.820194 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43977
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:38.925335 2952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:38.925616 2952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:38.925648 2952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:38.927084 2952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:38.927147 2952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:38.927181 2952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:49:38.929042 2952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:38.929363 2952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:38.929643 2952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:38.935768 2960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:38.935768 2958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:38.935833 2962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:38.936111 2952 server_base.cc:1034] running on GCE node
I20260807 22:49:38.936334 2952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:38.936666 2952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:38.937856 2952 hybrid_clock.cc:648] HybridClock initialized: now 1786142978937845 us; error 37 us; skew 500 ppm
Aug 07 22:49:38 dist-test-slave-rgb1 krb5kdc[2843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142978, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:38.940934 2952 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:49:38.941455 2952 webserver.cc:460] Webserver started at http://127.16.98.129:38279/ using document root <none> and password file <none>
I20260807 22:49:38.941728 2952 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:38.941798 2952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:38.941984 2952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:38.943099 2952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "bb83cde4e718446e8809a279402640db"
format_stamp: "Formatted at 2026-08-07 22:49:38 on dist-test-slave-rgb1"
server_key: "421c9d717d8f47beb220baf2cc25d41c"
server_key_iv: "10b01abaf7f8ef37ede8351e35c028fd"
server_key_version: "encryptionkey@0"
I20260807 22:49:38.943552 2952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "bb83cde4e718446e8809a279402640db"
format_stamp: "Formatted at 2026-08-07 22:49:38 on dist-test-slave-rgb1"
server_key: "421c9d717d8f47beb220baf2cc25d41c"
server_key_iv: "10b01abaf7f8ef37ede8351e35c028fd"
server_key_version: "encryptionkey@0"
I20260807 22:49:38.945220 2952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260807 22:49:38.946199 2969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:38.946545 2952 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:38.946712 2952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "bb83cde4e718446e8809a279402640db"
format_stamp: "Formatted at 2026-08-07 22:49:38 on dist-test-slave-rgb1"
server_key: "421c9d717d8f47beb220baf2cc25d41c"
server_key_iv: "10b01abaf7f8ef37ede8351e35c028fd"
server_key_version: "encryptionkey@0"
I20260807 22:49:38.946820 2952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:38.969580 2952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:38.972790 2952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:38.973209 2952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:38.973630 2952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:38.974118 2952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:38.974223 2952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:38.974265 2952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:38.974313 2952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:38.981045 2952 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:39791
I20260807 22:49:38.981066 3083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:39791 every 8 connection(s)
I20260807 22:49:38.981438 2952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
Aug 07 22:49:38 dist-test-slave-rgb1 krb5kdc[2843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142978, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:49:38.988484 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 2952
I20260807 22:49:38.988716 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:49:38.988999 16778 external_mini_cluster.cc:1428] Setting key 6836b75b57a56d94980a90d8e60ffe36
I20260807 22:49:38.993594 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:38.982471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:52625 (local address 127.16.98.190:43977)
0807 22:49:38.982574 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:38.982576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:38.982792 (+ 216us) server_negotiation.cc:407] Connection header received
0807 22:49:38.983283 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:38.983284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:38.983295 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:38.983362 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:38.984449 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:38.985100 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:38.986177 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:38.986393 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:38.986457 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:38.989036 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:38.989038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:38.989038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:38.989049 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:38.991186 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:38.991848 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:38.991849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:38.991850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:38.991933 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:38.992312 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:38.992313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:38.992314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:38.992462 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:38.992559 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:38.993329 (+ 770us) server_negotiation.cc:299] Negotiation successful
0807 22:49:38.993488 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 22:49:38.993760 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:38.982373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43977 (local address 127.16.98.129:52625)
0807 22:49:38.982638 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:38.982681 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:38.983183 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:38.983444 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:38.983448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:38.984115 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:38.984319 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:38.984324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:38.985213 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:38.985214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:38.986089 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:38.986091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:38.986485 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:38.986487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:38.986601 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:38.987219 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:49:38.987228 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:38.988920 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:38.991290 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:38.991299 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:38.991301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:38.991618 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:38.992068 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:38.992070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:38.992071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:38.992228 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:38.992588 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:38.992597 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:38.992987 (+ 390us) client_negotiation.cc:769] Sending connection context
0807 22:49:38.993285 (+ 298us) client_negotiation.cc:240] Negotiation successful
0807 22:49:38.993500 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":184,"threads_started":1}
I20260807 22:49:38.994278 3084 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43977
I20260807 22:49:38.994379 3084 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:38.994598 3084 heartbeater.cc:511] Master 127.16.98.190:43977 requested a full tablet report, sending...
I20260807 22:49:38.995187 2880 ts_manager.cc:194] Registered new tserver with Master: bb83cde4e718446e8809a279402640db (127.16.98.129:39791)
I20260807 22:49:38.995836 2880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:52625
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:49:39.026877 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43977
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:39.114466 3089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:39.114697 3089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:39.114729 3089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:39.115899 3089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:39.115944 3089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:39.115971 3089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:49:39.117513 3089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:39.117733 3089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:39.117938 3089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:39.123759 3096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:39.123756 3098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:39.123756 3095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:39.124100 3089 server_base.cc:1034] running on GCE node
I20260807 22:49:39.124364 3089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:39.124706 3089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:39.125923 3089 hybrid_clock.cc:648] HybridClock initialized: now 1786142979125915 us; error 39 us; skew 500 ppm
Aug 07 22:49:39 dist-test-slave-rgb1 krb5kdc[2843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142979, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:39.133114 3089 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:49:39.133759 3089 webserver.cc:460] Webserver started at http://127.16.98.130:40581/ using document root <none> and password file <none>
I20260807 22:49:39.134055 3089 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:39.134136 3089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:39.134320 3089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:39.135607 3089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57"
format_stamp: "Formatted at 2026-08-07 22:49:39 on dist-test-slave-rgb1"
server_key: "2e549fbe9452058fe7fb24c60083d3ee"
server_key_iv: "034919953cf9de7a4137d6caa281191c"
server_key_version: "encryptionkey@0"
I20260807 22:49:39.136214 3089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57"
format_stamp: "Formatted at 2026-08-07 22:49:39 on dist-test-slave-rgb1"
server_key: "2e549fbe9452058fe7fb24c60083d3ee"
server_key_iv: "034919953cf9de7a4137d6caa281191c"
server_key_version: "encryptionkey@0"
I20260807 22:49:39.138015 3089 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 22:49:39.139050 3105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:39.139302 3089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:39.139448 3089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57"
format_stamp: "Formatted at 2026-08-07 22:49:39 on dist-test-slave-rgb1"
server_key: "2e549fbe9452058fe7fb24c60083d3ee"
server_key_iv: "034919953cf9de7a4137d6caa281191c"
server_key_version: "encryptionkey@0"
I20260807 22:49:39.139586 3089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:39.151626 3089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:39.154184 3089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:39.154309 3089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:39.154489 3089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:39.154806 3089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:39.154841 3089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:39.154865 3089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:39.154913 3089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:39.161036 3089 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:41943
I20260807 22:49:39.161134 3221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:41943 every 8 connection(s)
I20260807 22:49:39.161466 3089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:49:39.163509 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 3089
I20260807 22:49:39.163659 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:49:39.163901 16778 external_mini_cluster.cc:1428] Setting key 047eb594be782fa5cdd10eec2aa9f9c4
Aug 07 22:49:39 dist-test-slave-rgb1 krb5kdc[2843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142979, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
I20260807 22:49:39.173556 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.162316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:38855 (local address 127.16.98.190:43977)
0807 22:49:39.162380 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:39.162382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:39.162576 (+ 194us) server_negotiation.cc:407] Connection header received
0807 22:49:39.163223 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:39.163225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:39.163238 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:39.163309 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:39.164679 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.165508 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.166888 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.167118 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.167202 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.170385 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:39.170386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:39.170387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:39.170396 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:39.171951 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:39.172405 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:39.172406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:39.172407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:39.172473 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:39.172778 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:39.172780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:39.172780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:39.172958 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:39.173039 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:39.173347 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 22:49:39.173458 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260807 22:49:39.173772 3225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.162271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43977 (local address 127.16.98.130:38855)
0807 22:49:39.162432 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:39.162481 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:39.163099 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:39.163357 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:39.163362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:39.164115 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:39.164422 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.164427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.165646 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.165647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.166758 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.166761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.167227 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.167228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.167383 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.168173 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:49:39.168182 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:39.170244 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:39.172063 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:39.172073 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:39.172076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:39.172321 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:39.172546 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:39.172547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:39.172548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:39.172706 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:39.173040 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:39.173049 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:39.173206 (+ 157us) client_negotiation.cc:769] Sending connection context
0807 22:49:39.173324 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 22:49:39.173510 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260807 22:49:39.174540 3222 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43977
I20260807 22:49:39.174690 3222 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:39.174908 3222 heartbeater.cc:511] Master 127.16.98.190:43977 requested a full tablet report, sending...
I20260807 22:49:39.175479 2880 ts_manager.cc:194] Registered new tserver with Master: 0a9f3ffb6ee9437eb9acfd02dc238b57 (127.16.98.130:41943)
I20260807 22:49:39.175920 2880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:38855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:49:39.196395 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43977
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:44143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:39.290855 3228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:39.291080 3228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:39.291105 3228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:39.292272 3228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:39.292320 3228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:39.292346 3228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:49:39.293924 3228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:44143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:39.294190 3228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:39.294461 3228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:39.300231 3236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:39.300269 3233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:39.300244 3228 server_base.cc:1034] running on GCE node
W20260807 22:49:39.300266 3234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:39.300675 3228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:39.300961 3228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:39.302147 3228 hybrid_clock.cc:648] HybridClock initialized: now 1786142979302130 us; error 29 us; skew 500 ppm
Aug 07 22:49:39 dist-test-slave-rgb1 krb5kdc[2843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142979, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:39.305148 3228 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:49:39.305744 3228 webserver.cc:460] Webserver started at http://127.16.98.131:44987/ using document root <none> and password file <none>
I20260807 22:49:39.306015 3228 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:39.306092 3228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:39.306387 3228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:39.307593 3228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "6f10f4cd37b549519c6e37a8823b41da"
format_stamp: "Formatted at 2026-08-07 22:49:39 on dist-test-slave-rgb1"
server_key: "be38dc6ee3afff32c053d6cf1e283d58"
server_key_iv: "76ade774c699bb8d07f857e0831307b7"
server_key_version: "encryptionkey@0"
I20260807 22:49:39.307988 3228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "6f10f4cd37b549519c6e37a8823b41da"
format_stamp: "Formatted at 2026-08-07 22:49:39 on dist-test-slave-rgb1"
server_key: "be38dc6ee3afff32c053d6cf1e283d58"
server_key_iv: "76ade774c699bb8d07f857e0831307b7"
server_key_version: "encryptionkey@0"
I20260807 22:49:39.309587 3228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260807 22:49:39.310622 3243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:39.311100 3228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:49:39.311245 3228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "6f10f4cd37b549519c6e37a8823b41da"
format_stamp: "Formatted at 2026-08-07 22:49:39 on dist-test-slave-rgb1"
server_key: "be38dc6ee3afff32c053d6cf1e283d58"
server_key_iv: "76ade774c699bb8d07f857e0831307b7"
server_key_version: "encryptionkey@0"
I20260807 22:49:39.311329 3228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:39.321552 3228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:39.324198 3228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:39.324419 3228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:39.324766 3228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:39.325112 3228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:39.325171 3228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:39.325215 3228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:39.325244 3228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:39.331247 3228 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:36553
I20260807 22:49:39.331288 3361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:36553 every 8 connection(s)
I20260807 22:49:39.331660 3228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:49:39.333173 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 3228
I20260807 22:49:39.333308 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:49:39.333519 16778 external_mini_cluster.cc:1428] Setting key 9412f644c985d518ea79fce534021772
Aug 07 22:49:39 dist-test-slave-rgb1 krb5kdc[2843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142979, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:49:39.342026 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.332530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:51243 (local address 127.16.98.190:43977)
0807 22:49:39.332663 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:39.332666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:39.332866 (+ 200us) server_negotiation.cc:407] Connection header received
0807 22:49:39.333493 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:39.333494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:39.333507 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:39.333555 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:39.334763 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.335286 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.336313 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.336506 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.336601 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.338956 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:39.338957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:39.338958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:39.338964 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:39.340408 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:39.341001 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:39.341002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:39.341003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:39.341073 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:39.341376 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:39.341377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:39.341377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:39.341529 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:39.341590 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:39.341842 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 22:49:39.341906 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 22:49:39.342253 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.332497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43977 (local address 127.16.98.131:51243)
0807 22:49:39.332721 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:39.332774 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:39.333372 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:39.333666 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:39.333672 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:39.334409 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:39.334669 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.334675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.335398 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.335399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.336209 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.336211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.336614 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.336616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.336742 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.337273 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:49:39.337281 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:39.338848 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:39.340514 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:39.340522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:39.340523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:39.340919 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:39.341150 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:39.341151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:39.341151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:39.341258 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:39.341598 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:39.341604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:39.341724 (+ 120us) client_negotiation.cc:769] Sending connection context
0807 22:49:39.341813 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 22:49:39.341982 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":127,"threads_started":1}
I20260807 22:49:39.343034 3362 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43977
I20260807 22:49:39.343142 3362 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:39.343343 3362 heartbeater.cc:511] Master 127.16.98.190:43977 requested a full tablet report, sending...
I20260807 22:49:39.343814 2880 ts_manager.cc:194] Registered new tserver with Master: 6f10f4cd37b549519c6e37a8823b41da (127.16.98.131:36553)
I20260807 22:49:39.344223 2880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:51243
I20260807 22:49:39.345476 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:49:39.354213 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.346947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.16.98.190:43977)
0807 22:49:39.347014 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:39.347016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:39.347215 (+ 199us) server_negotiation.cc:407] Connection header received
0807 22:49:39.347255 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:39.347256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:39.347265 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:39.347317 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:39.348136 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.348681 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.349446 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.349625 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.349682 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.351265 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:39.351266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:39.351267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:39.351273 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:39.352880 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:39.353321 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:39.353322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:39.353322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:39.353386 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:39.353660 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:39.353660 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:39.353660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:39.353792 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:39.353946 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:39.354019 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 22:49:39.354074 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 22:49:39.356242 2880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:49:39.358006 2880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:49:39.364928 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.360618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:41943 (local address 127.0.0.1:51182)
0807 22:49:39.361092 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:39.361104 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:39.361151 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:39.361332 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:39.361335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:39.361338 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:39.361500 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.361503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.362472 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.362473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.363608 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.363610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.364526 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.364527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.364675 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.364688 (+ 13us) client_negotiation.cc:769] Sending connection context
0807 22:49:39.364785 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 22:49:39.364846 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":90,"threads_started":1}
I20260807 22:49:39.364950 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.360747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51182 (local address 127.16.98.130:41943)
0807 22:49:39.361152 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:39.361157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:39.361184 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:49:39.361228 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:39.361231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:39.361247 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:39.361325 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:39.361593 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.362356 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.363709 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.364434 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.364527 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.364615 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:39.364806 (+ 191us) server_negotiation.cc:299] Negotiation successful
0807 22:49:39.364851 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":96,"threads_started":1}
I20260807 22:49:39.365235 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.360852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:36553 (local address 127.0.0.1:39792)
0807 22:49:39.361327 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:39.361339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:39.361397 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:39.361682 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:39.361682 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:39.361684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:39.361719 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.361720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.363008 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.363009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.363939 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.363941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.364994 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.364995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.365063 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.365067 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:49:39.365120 (+ 53us) client_negotiation.cc:240] Negotiation successful
0807 22:49:39.365161 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":24,"thread_start_us":172,"threads_started":1}
I20260807 22:49:39.365283 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.360932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.16.98.131:36553)
0807 22:49:39.361261 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:39.361265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:39.361463 (+ 198us) server_negotiation.cc:407] Connection header received
0807 22:49:39.361508 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:39.361510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:39.361524 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:39.361594 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:39.361809 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.362893 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.364033 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.364825 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.364898 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.364953 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:39.365156 (+ 203us) server_negotiation.cc:299] Negotiation successful
0807 22:49:39.365208 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":94,"threads_started":1}
I20260807 22:49:39.366106 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.360613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:39791 (local address 127.0.0.1:36040)
0807 22:49:39.360772 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:39.360818 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:39.360892 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:39.361503 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:39.361504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:39.361505 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:39.361577 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.361580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.363010 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.363011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.364151 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.364153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.365867 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.365868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.365952 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.365956 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:49:39.365988 (+ 32us) client_negotiation.cc:240] Negotiation successful
0807 22:49:39.366028 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":25,"thread_start_us":69,"threads_started":1}
I20260807 22:49:39.366132 3155 tablet_service.cc:1467] Processing CreateTablet for tablet de5c3816e6af4f308edd61c494ef2be9 (DEFAULT_TABLE table=test-table [id=c971b751f4ce4668811bbd45b6024bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:39.366262 3293 tablet_service.cc:1467] Processing CreateTablet for tablet de5c3816e6af4f308edd61c494ef2be9 (DEFAULT_TABLE table=test-table [id=c971b751f4ce4668811bbd45b6024bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:39.366432 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.361125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.16.98.129:39791)
0807 22:49:39.361305 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:39.361310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:39.361322 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:49:39.361367 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:39.361369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:39.361385 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:39.361653 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:39.361724 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.362875 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.364248 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.365008 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.366207 (+ 1199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.366278 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:39.366306 (+ 28us) server_negotiation.cc:299] Negotiation successful
0807 22:49:39.366353 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260807 22:49:39.366532 3155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5c3816e6af4f308edd61c494ef2be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:39.366647 3293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5c3816e6af4f308edd61c494ef2be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:39.367129 3017 tablet_service.cc:1467] Processing CreateTablet for tablet de5c3816e6af4f308edd61c494ef2be9 (DEFAULT_TABLE table=test-table [id=c971b751f4ce4668811bbd45b6024bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:39.367429 3017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5c3816e6af4f308edd61c494ef2be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:39.370074 3380 tablet_bootstrap.cc:492] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57: Bootstrap starting.
I20260807 22:49:39.370823 3380 tablet_bootstrap.cc:654] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:39.371045 3381 tablet_bootstrap.cc:492] T de5c3816e6af4f308edd61c494ef2be9 P bb83cde4e718446e8809a279402640db: Bootstrap starting.
I20260807 22:49:39.371172 3379 tablet_bootstrap.cc:492] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da: Bootstrap starting.
I20260807 22:49:39.371507 3380 log.cc:824] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:39.371747 3381 tablet_bootstrap.cc:654] T de5c3816e6af4f308edd61c494ef2be9 P bb83cde4e718446e8809a279402640db: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:39.371797 3379 tablet_bootstrap.cc:654] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:39.372364 3380 tablet_bootstrap.cc:492] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57: No bootstrap required, opened a new log
I20260807 22:49:39.372388 3381 log.cc:824] T de5c3816e6af4f308edd61c494ef2be9 P bb83cde4e718446e8809a279402640db: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:39.372388 3379 log.cc:824] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:39.372453 3380 ts_tablet_manager.cc:1397] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:49:39.373116 3381 tablet_bootstrap.cc:492] T de5c3816e6af4f308edd61c494ef2be9 P bb83cde4e718446e8809a279402640db: No bootstrap required, opened a new log
I20260807 22:49:39.373126 3379 tablet_bootstrap.cc:492] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da: No bootstrap required, opened a new log
I20260807 22:49:39.373185 3381 ts_tablet_manager.cc:1397] T de5c3816e6af4f308edd61c494ef2be9 P bb83cde4e718446e8809a279402640db: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260807 22:49:39.373185 3379 ts_tablet_manager.cc:1397] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260807 22:49:39.374238 3380 raft_consensus.cc:348] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 41943 } } peers { permanent_uuid: "bb83cde4e718446e8809a279402640db" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39791 } } peers { permanent_uuid: "6f10f4cd37b549519c6e37a8823b41da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 36553 } }
I20260807 22:49:39.374442 3380 raft_consensus.cc:374] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:39.374492 3380 raft_consensus.cc:729] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a9f3ffb6ee9437eb9acfd02dc238b57, State: Initialized, Role: FOLLOWER
I20260807 22:49:39.374626 3380 consensus_queue.cc:260] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 41943 } } peers { permanent_uuid: "bb83cde4e718446e8809a279402640db" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39791 } } peers { permanent_uuid: "6f10f4cd37b549519c6e37a8823b41da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 36553 } }
I20260807 22:49:39.374945 3222 heartbeater.cc:503] Master 127.16.98.190:43977 was elected leader, sending a full tablet report...
I20260807 22:49:39.375277 3380 ts_tablet_manager.cc:1428] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:49:39.375182 3381 raft_consensus.cc:348] T de5c3816e6af4f308edd61c494ef2be9 P bb83cde4e718446e8809a279402640db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 41943 } } peers { permanent_uuid: "bb83cde4e718446e8809a279402640db" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39791 } } peers { permanent_uuid: "6f10f4cd37b549519c6e37a8823b41da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 36553 } }
I20260807 22:49:39.375182 3379 raft_consensus.cc:348] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 41943 } } peers { permanent_uuid: "bb83cde4e718446e8809a279402640db" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39791 } } peers { permanent_uuid: "6f10f4cd37b549519c6e37a8823b41da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 36553 } }
I20260807 22:49:39.375341 3381 raft_consensus.cc:374] T de5c3816e6af4f308edd61c494ef2be9 P bb83cde4e718446e8809a279402640db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:39.375341 3379 raft_consensus.cc:374] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:39.375378 3381 raft_consensus.cc:729] T de5c3816e6af4f308edd61c494ef2be9 P bb83cde4e718446e8809a279402640db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb83cde4e718446e8809a279402640db, State: Initialized, Role: FOLLOWER
I20260807 22:49:39.375378 3379 raft_consensus.cc:729] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f10f4cd37b549519c6e37a8823b41da, State: Initialized, Role: FOLLOWER
I20260807 22:49:39.375497 3381 consensus_queue.cc:260] T de5c3816e6af4f308edd61c494ef2be9 P bb83cde4e718446e8809a279402640db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 41943 } } peers { permanent_uuid: "bb83cde4e718446e8809a279402640db" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39791 } } peers { permanent_uuid: "6f10f4cd37b549519c6e37a8823b41da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 36553 } }
I20260807 22:49:39.375496 3379 consensus_queue.cc:260] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 41943 } } peers { permanent_uuid: "bb83cde4e718446e8809a279402640db" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39791 } } peers { permanent_uuid: "6f10f4cd37b549519c6e37a8823b41da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 36553 } }
I20260807 22:49:39.375773 3381 ts_tablet_manager.cc:1428] T de5c3816e6af4f308edd61c494ef2be9 P bb83cde4e718446e8809a279402640db: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 22:49:39.375773 3379 ts_tablet_manager.cc:1428] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 22:49:39.375890 3362 heartbeater.cc:503] Master 127.16.98.190:43977 was elected leader, sending a full tablet report...
I20260807 22:49:39.375932 3084 heartbeater.cc:503] Master 127.16.98.190:43977 was elected leader, sending a full tablet report...
I20260807 22:49:39.409891 3387 raft_consensus.cc:482] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:49:39.410038 3387 raft_consensus.cc:504] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 41943 } } peers { permanent_uuid: "bb83cde4e718446e8809a279402640db" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39791 } } peers { permanent_uuid: "6f10f4cd37b549519c6e37a8823b41da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 36553 } }
I20260807 22:49:39.410372 3387 leader_election.cc:288] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a9f3ffb6ee9437eb9acfd02dc238b57 (127.16.98.130:41943), bb83cde4e718446e8809a279402640db (127.16.98.129:39791)
W20260807 22:49:39.412072 3223 tablet.cc:2369] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:49:39.413807 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.410705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:40931 (local address 127.16.98.129:39791)
0807 22:49:39.410783 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:39.410786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:39.410935 (+ 149us) server_negotiation.cc:407] Connection header received
0807 22:49:39.411076 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:39.411077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:39.411092 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:39.411154 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:39.411339 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.411797 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.412979 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.413415 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.413474 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.413483 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:39.413664 (+ 181us) server_negotiation.cc:299] Negotiation successful
0807 22:49:39.413706 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 22:49:39.413815 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.410599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:39791 (local address 127.16.98.131:40931)
0807 22:49:39.410854 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:39.410869 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:39.410932 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:39.411172 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:39.411173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:39.411179 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:39.411260 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.411262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.411887 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.411888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.412887 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.412889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.413499 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.413501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.413583 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.413588 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:49:39.413651 (+ 63us) client_negotiation.cc:240] Negotiation successful
0807 22:49:39.413719 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":137,"threads_started":1}
I20260807 22:49:39.413954 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.410795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:53291 (local address 127.16.98.130:41943)
0807 22:49:39.410869 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:39.410872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:39.410903 (+ 31us) server_negotiation.cc:407] Connection header received
0807 22:49:39.410997 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:39.410998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:39.411011 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:39.411076 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:39.411247 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.411864 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.413082 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.413544 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.413629 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.413640 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:39.413814 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 22:49:39.413880 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 22:49:39.414111 3388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.410585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:41943 (local address 127.16.98.131:53291)
0807 22:49:39.410830 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:39.410841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:39.410932 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:39.411078 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:39.411079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:39.411081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:39.411167 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.411169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.411992 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.411993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.412970 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.412973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.413653 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.413655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.413726 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.413730 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:49:39.413789 (+ 59us) client_negotiation.cc:240] Negotiation successful
0807 22:49:39.413830 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260807 22:49:39.414322 3038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5c3816e6af4f308edd61c494ef2be9" candidate_uuid: "6f10f4cd37b549519c6e37a8823b41da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb83cde4e718446e8809a279402640db" is_pre_election: true
I20260807 22:49:39.414534 3038 raft_consensus.cc:2454] T de5c3816e6af4f308edd61c494ef2be9 P bb83cde4e718446e8809a279402640db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f10f4cd37b549519c6e37a8823b41da in term 0.
I20260807 22:49:39.414536 3175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5c3816e6af4f308edd61c494ef2be9" candidate_uuid: "6f10f4cd37b549519c6e37a8823b41da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57" is_pre_election: true
I20260807 22:49:39.414669 3175 raft_consensus.cc:2454] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f10f4cd37b549519c6e37a8823b41da in term 0.
I20260807 22:49:39.414813 3246 leader_election.cc:302] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f10f4cd37b549519c6e37a8823b41da, bb83cde4e718446e8809a279402640db; no voters:
I20260807 22:49:39.415028 3387 raft_consensus.cc:2781] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:49:39.415102 3387 raft_consensus.cc:482] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:49:39.415128 3387 raft_consensus.cc:3037] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:39.415864 3387 raft_consensus.cc:504] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 41943 } } peers { permanent_uuid: "bb83cde4e718446e8809a279402640db" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39791 } } peers { permanent_uuid: "6f10f4cd37b549519c6e37a8823b41da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 36553 } }
I20260807 22:49:39.416042 3387 leader_election.cc:288] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [CANDIDATE]: Term 1 election: Requested vote from peers 0a9f3ffb6ee9437eb9acfd02dc238b57 (127.16.98.130:41943), bb83cde4e718446e8809a279402640db (127.16.98.129:39791)
I20260807 22:49:39.416209 3175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5c3816e6af4f308edd61c494ef2be9" candidate_uuid: "6f10f4cd37b549519c6e37a8823b41da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57"
I20260807 22:49:39.416205 3038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5c3816e6af4f308edd61c494ef2be9" candidate_uuid: "6f10f4cd37b549519c6e37a8823b41da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb83cde4e718446e8809a279402640db"
I20260807 22:49:39.416294 3175 raft_consensus.cc:3037] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:39.416318 3038 raft_consensus.cc:3037] T de5c3816e6af4f308edd61c494ef2be9 P bb83cde4e718446e8809a279402640db [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:39.417110 3175 raft_consensus.cc:2454] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f10f4cd37b549519c6e37a8823b41da in term 1.
I20260807 22:49:39.417299 3244 leader_election.cc:302] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a9f3ffb6ee9437eb9acfd02dc238b57, 6f10f4cd37b549519c6e37a8823b41da; no voters:
I20260807 22:49:39.417409 3387 raft_consensus.cc:2781] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:39.417438 3038 raft_consensus.cc:2454] T de5c3816e6af4f308edd61c494ef2be9 P bb83cde4e718446e8809a279402640db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f10f4cd37b549519c6e37a8823b41da in term 1.
I20260807 22:49:39.417610 3387 raft_consensus.cc:686] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [term 1 LEADER]: Becoming Leader. State: Replica: 6f10f4cd37b549519c6e37a8823b41da, State: Running, Role: LEADER
I20260807 22:49:39.417865 3387 consensus_queue.cc:237] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 41943 } } peers { permanent_uuid: "bb83cde4e718446e8809a279402640db" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39791 } } peers { permanent_uuid: "6f10f4cd37b549519c6e37a8823b41da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 36553 } }
I20260807 22:49:39.419035 2888 catalog_manager.cc:5495] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f10f4cd37b549519c6e37a8823b41da (127.16.98.131). New cstate: current_term: 1 leader_uuid: "6f10f4cd37b549519c6e37a8823b41da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 41943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb83cde4e718446e8809a279402640db" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f10f4cd37b549519c6e37a8823b41da" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 36553 } health_report { overall_health: HEALTHY } } }
I20260807 22:49:39.425654 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.422989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39796 (local address 127.16.98.131:36553)
0807 22:49:39.423083 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:39.423085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:39.423094 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:49:39.423125 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:39.423126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:39.423143 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:39.423245 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:49:39.423454 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.423964 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.424808 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.425028 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.425103 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.425260 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:49:39.425366 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:49:39.425482 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:39.425515 (+ 33us) server_negotiation.cc:299] Negotiation successful
0807 22:49:39.425527 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 22:49:39.427834 3175 raft_consensus.cc:1261] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57 [term 1 FOLLOWER]: Refusing update from remote peer 6f10f4cd37b549519c6e37a8823b41da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:49:39.427834 3038 raft_consensus.cc:1261] T de5c3816e6af4f308edd61c494ef2be9 P bb83cde4e718446e8809a279402640db [term 1 FOLLOWER]: Refusing update from remote peer 6f10f4cd37b549519c6e37a8823b41da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:49:39.428254 3387 consensus_queue.cc:1035] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb83cde4e718446e8809a279402640db" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 39791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:49:39.428413 3389 consensus_queue.cc:1035] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a9f3ffb6ee9437eb9acfd02dc238b57" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 41943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:49:39.431161 3396 mvcc.cc:205] Tried to move back new op lower bound from 7316041643732983808 to 7316041643696496640. Current Snapshot: MvccSnapshot[applied={T|T < 7316041643732983808}]
I20260807 22:49:39.434139 2888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:49:39.434263 2888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:49:39.435390 2888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96186599841a458197c8139470203ab7: Sending DeleteTablet for 3 replicas of tablet de5c3816e6af4f308edd61c494ef2be9
I20260807 22:49:39.435782 3155 tablet_service.cc:1514] Processing DeleteTablet for tablet de5c3816e6af4f308edd61c494ef2be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:39 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:51182
I20260807 22:49:39.435806 3293 tablet_service.cc:1514] Processing DeleteTablet for tablet de5c3816e6af4f308edd61c494ef2be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:39 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:39792
I20260807 22:49:39.435848 3017 tablet_service.cc:1514] Processing DeleteTablet for tablet de5c3816e6af4f308edd61c494ef2be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:39 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:36040
I20260807 22:49:39.436275 3404 tablet_replica.cc:330] T de5c3816e6af4f308edd61c494ef2be9 P bb83cde4e718446e8809a279402640db: stopping tablet replica
I20260807 22:49:39.436378 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 2952
I20260807 22:49:39.438112 3405 tablet_replica.cc:330] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da: stopping tablet replica
I20260807 22:49:39.438252 3405 raft_consensus.cc:2229] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:49:39.438405 3405 raft_consensus.cc:2258] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:49:39.438925 3405 ts_tablet_manager.cc:1905] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:49:39.440742 3403 tablet_replica.cc:330] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57: stopping tablet replica
I20260807 22:49:39.440944 3403 raft_consensus.cc:2229] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:49:39.441133 3403 raft_consensus.cc:2258] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:49:39.441543 3405 ts_tablet_manager.cc:1918] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:49:39.441634 3405 log.cc:1196] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/de5c3816e6af4f308edd61c494ef2be9
I20260807 22:49:39.441973 3405 ts_tablet_manager.cc:1939] T de5c3816e6af4f308edd61c494ef2be9 P 6f10f4cd37b549519c6e37a8823b41da: Deleting consensus metadata
I20260807 22:49:39.442490 2873 catalog_manager.cc:4841] TS 6f10f4cd37b549519c6e37a8823b41da (127.16.98.131:36553): tablet de5c3816e6af4f308edd61c494ef2be9 (table test-table [id=c971b751f4ce4668811bbd45b6024bf2]) successfully deleted
W20260807 22:49:39.444591 2875 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:39791 (error 108)
I20260807 22:49:39.445364 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.445212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:39791 (local address 127.0.0.1:36042)
0807 22:49:39.445276 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:39.445312 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:39791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260807 22:49:39.445489 2875 catalog_manager.cc:4584] TS bb83cde4e718446e8809a279402640db (127.16.98.129:39791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de5c3816e6af4f308edd61c494ef2be9: Network error: Client connection negotiation failed: client connection to 127.16.98.129:39791: connect: Connection refused (error 111)
I20260807 22:49:39.445585 3403 ts_tablet_manager.cc:1905] T de5c3816e6af4f308edd61c494ef2be9 P 0a9f3ffb6ee9437eb9acfd02dc238b57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:49:39.445922 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 3089
I20260807 22:49:39.453855 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 3228
I20260807 22:49:39.453877 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.453737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:41943 (local address 127.0.0.1:51196)
0807 22:49:39.453802 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:39.453839 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.130:41943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260807 22:49:39.459673 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 2856
2026-08-07T22:49:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2133 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-07T22:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:39Z Disabled control of system clock
I20260807 22:49:39.492432 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43279
--webserver_interface=127.16.98.190
--webserver_port=0
--builtin_ntp_servers=127.16.98.148:34247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:43279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 22:49:39.585549 3410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:39.585843 3410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:39.585887 3410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:39.587146 3410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:39.587220 3410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:39.587244 3410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:39.587273 3410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:39.587306 3410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:39.588749 3410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:43279
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:39.589041 3410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:39.589419 3410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:39.594866 3415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:39.594980 3418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:39.594866 3416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:39.595094 3410 server_base.cc:1034] running on GCE node
I20260807 22:49:39.595530 3410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:39.595878 3410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:39.597075 3410 hybrid_clock.cc:648] HybridClock initialized: now 1786142979597045 us; error 49 us; skew 500 ppm
I20260807 22:49:39.598479 3410 webserver.cc:460] Webserver started at http://127.16.98.190:43787/ using document root <none> and password file <none>
I20260807 22:49:39.598762 3410 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:39.598836 3410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:39.599031 3410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:39.600188 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "277f524c51fc41c7bf102a116a7cb715"
format_stamp: "Formatted at 2026-08-07 22:49:39 on dist-test-slave-rgb1"
server_key: "0cd5903eb9c1bddf0f99a8e6748402f7"
server_key_iv: "1a6b5d2a5636f72c954cd0a9cb5e6e39"
server_key_version: "encryptionkey@0"
I20260807 22:49:39.600592 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "277f524c51fc41c7bf102a116a7cb715"
format_stamp: "Formatted at 2026-08-07 22:49:39 on dist-test-slave-rgb1"
server_key: "0cd5903eb9c1bddf0f99a8e6748402f7"
server_key_iv: "1a6b5d2a5636f72c954cd0a9cb5e6e39"
server_key_version: "encryptionkey@0"
I20260807 22:49:39.601999 3410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:39.602886 3425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:39.603125 3410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:39.603230 3410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "277f524c51fc41c7bf102a116a7cb715"
format_stamp: "Formatted at 2026-08-07 22:49:39 on dist-test-slave-rgb1"
server_key: "0cd5903eb9c1bddf0f99a8e6748402f7"
server_key_iv: "1a6b5d2a5636f72c954cd0a9cb5e6e39"
server_key_version: "encryptionkey@0"
I20260807 22:49:39.603312 3410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:39.614059 3410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:39.614449 3410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:39.614609 3410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:39.619024 3410 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:43279
I20260807 22:49:39.619064 3479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:43279 every 8 connection(s)
I20260807 22:49:39.619441 3410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:39.620549 3480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:39.623047 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715: Bootstrap starting.
I20260807 22:49:39.623683 3480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:39.624009 3480 log.cc:824] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:39.624866 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715: No bootstrap required, opened a new log
I20260807 22:49:39.626605 3480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277f524c51fc41c7bf102a116a7cb715" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43279 } }
I20260807 22:49:39.626756 3480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:39.626781 3480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277f524c51fc41c7bf102a116a7cb715, State: Initialized, Role: FOLLOWER
I20260807 22:49:39.626960 3480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277f524c51fc41c7bf102a116a7cb715" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43279 } }
I20260807 22:49:39.627028 3480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:39.627050 3480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:39.627077 3480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:39.627667 3480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277f524c51fc41c7bf102a116a7cb715" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43279 } }
I20260807 22:49:39.627754 3480 leader_election.cc:302] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 277f524c51fc41c7bf102a116a7cb715; no voters:
I20260807 22:49:39.627997 3480 leader_election.cc:288] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:39.628122 3483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:39.628278 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 3410
I20260807 22:49:39.628304 3480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:39.628324 3483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [term 1 LEADER]: Becoming Leader. State: Replica: 277f524c51fc41c7bf102a116a7cb715, State: Running, Role: LEADER
I20260807 22:49:39.628402 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:39.628468 3483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277f524c51fc41c7bf102a116a7cb715" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43279 } }
I20260807 22:49:39.628688 16778 external_mini_cluster.cc:1428] Setting key 26ffba1493eb97f525b382cc5eae28dd
I20260807 22:49:39.629005 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "277f524c51fc41c7bf102a116a7cb715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277f524c51fc41c7bf102a116a7cb715" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43279 } } }
I20260807 22:49:39.629101 3485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 277f524c51fc41c7bf102a116a7cb715. Latest consensus state: current_term: 1 leader_uuid: "277f524c51fc41c7bf102a116a7cb715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277f524c51fc41c7bf102a116a7cb715" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43279 } } }
I20260807 22:49:39.629122 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:39.629365 3485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:39.629658 3494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:39.630275 3494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:39.631973 3494 catalog_manager.cc:1347] Generated new cluster ID: 9ebc7a62a4f74f719ddc1576e4fbd093
I20260807 22:49:39.632033 3494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:49:39.635556 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.629446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.16.98.190:43279)
0807 22:49:39.629670 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:39.629681 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:39.629801 (+ 120us) server_negotiation.cc:407] Connection header received
0807 22:49:39.630144 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:39.630153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:39.630238 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:39.630734 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:39.630859 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.632380 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.633310 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.633800 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.633974 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.634832 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:39.634836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:39.634836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:39.635042 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:39.635078 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:39.635090 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:39.635175 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:39.635260 (+ 85us) server_negotiation.cc:299] Negotiation successful
0807 22:49:39.635352 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260807 22:49:39.672242 3494 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:39.673118 3494 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:39.676226 3494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715: Generated new TSK 0
I20260807 22:49:39.676447 3494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:49:39.683074 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43279
--builtin_ntp_servers=127.16.98.148:34247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 22:49:39.792974 3505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:39.793182 3505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:39.793208 3505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:39.794386 3505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:39.794430 3505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:39.794457 3505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:49:39.795868 3505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:39.796145 3505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:39.796478 3505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:39.803432 3513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:39.803380 3510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:39.803640 3505 server_base.cc:1034] running on GCE node
W20260807 22:49:39.803396 3511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:39.804113 3505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:39.804415 3505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:39.805886 3505 hybrid_clock.cc:648] HybridClock initialized: now 1786142979805838 us; error 48 us; skew 500 ppm
I20260807 22:49:39.807875 3505 webserver.cc:460] Webserver started at http://127.16.98.129:35745/ using document root <none> and password file <none>
I20260807 22:49:39.808166 3505 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:39.808241 3505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:39.808396 3505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:39.809801 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "5c1bd5b06a82496cb507d2394197c19a"
format_stamp: "Formatted at 2026-08-07 22:49:39 on dist-test-slave-rgb1"
server_key: "c2965d816620f3ebe034c17f2a473180"
server_key_iv: "38a1dfbf7d4a0ffc909cf00867285dd2"
server_key_version: "encryptionkey@0"
I20260807 22:49:39.810326 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "5c1bd5b06a82496cb507d2394197c19a"
format_stamp: "Formatted at 2026-08-07 22:49:39 on dist-test-slave-rgb1"
server_key: "c2965d816620f3ebe034c17f2a473180"
server_key_iv: "38a1dfbf7d4a0ffc909cf00867285dd2"
server_key_version: "encryptionkey@0"
I20260807 22:49:39.811944 3505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:39.813432 3521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:39.813663 3505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:49:39.813779 3505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "5c1bd5b06a82496cb507d2394197c19a"
format_stamp: "Formatted at 2026-08-07 22:49:39 on dist-test-slave-rgb1"
server_key: "c2965d816620f3ebe034c17f2a473180"
server_key_iv: "38a1dfbf7d4a0ffc909cf00867285dd2"
server_key_version: "encryptionkey@0"
I20260807 22:49:39.813906 3505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:39.823846 3505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:39.824185 3505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:39.824342 3505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:39.824644 3505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:39.824999 3505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:39.825055 3505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:39.825105 3505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:39.825134 3505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:39.831372 3505 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:44625
I20260807 22:49:39.831545 3636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:44625 every 8 connection(s)
I20260807 22:49:39.831820 3505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:49:39.837570 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.832707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:58993 (local address 127.16.98.190:43279)
0807 22:49:39.832786 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:39.832788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:39.833072 (+ 284us) server_negotiation.cc:407] Connection header received
0807 22:49:39.833591 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:39.833592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:39.833610 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:39.833683 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:39.834359 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.834877 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.835934 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.836213 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.836289 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.837271 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:39.837273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:39.837273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:39.837278 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:39.837293 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:39.837297 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:39.837367 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:39.837469 (+ 102us) server_negotiation.cc:299] Negotiation successful
0807 22:49:39.837505 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 22:49:39.837740 3639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.832684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43279 (local address 127.16.98.129:58993)
0807 22:49:39.832920 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:39.832958 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:39.833476 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:39.833737 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:39.833740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:39.833749 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:39.834268 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.834272 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.835013 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.835015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.835832 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.835834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.836312 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.836313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.836430 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.837112 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:49:39.837113 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:39.837142 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:49:39.837167 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:39.837370 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:39.837373 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:39.837375 (+ 2us) client_negotiation.cc:769] Sending connection context
0807 22:49:39.837423 (+ 48us) client_negotiation.cc:240] Negotiation successful
0807 22:49:39.837508 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":112,"threads_started":1}
I20260807 22:49:39.838374 3637 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43279
I20260807 22:49:39.838495 3637 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:39.838743 3637 heartbeater.cc:511] Master 127.16.98.190:43279 requested a full tablet report, sending...
I20260807 22:49:39.839479 3443 ts_manager.cc:194] Registered new tserver with Master: 5c1bd5b06a82496cb507d2394197c19a (127.16.98.129:44625)
I20260807 22:49:39.840186 3443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.98.129:58993
I20260807 22:49:39.840778 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 3505
I20260807 22:49:39.840911 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:49:39.841152 16778 external_mini_cluster.cc:1428] Setting key e8bc77ab4c0ad9c1ca1eeb55006d1baa
I20260807 22:49:39.842226 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43279
--builtin_ntp_servers=127.16.98.148:34247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 22:49:39.933005 3640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:39.933243 3640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:39.933267 3640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:39.934501 3640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:39.934540 3640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:39.934568 3640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:49:39.935948 3640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:39.936156 3640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:39.936442 3640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:39.941855 3646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:39.941867 3645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:39.941861 3648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:39.942375 3640 server_base.cc:1034] running on GCE node
I20260807 22:49:39.942600 3640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:39.942817 3640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:39.944020 3640 hybrid_clock.cc:648] HybridClock initialized: now 1786142979944005 us; error 30 us; skew 500 ppm
I20260807 22:49:39.945446 3640 webserver.cc:460] Webserver started at http://127.16.98.130:35447/ using document root <none> and password file <none>
I20260807 22:49:39.945678 3640 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:39.945724 3640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:39.945819 3640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:39.946869 3640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "a0d73308ec86426aaea9b1d375995e71"
format_stamp: "Formatted at 2026-08-07 22:49:39 on dist-test-slave-rgb1"
server_key: "448f64b818372f16fccedde0ecf8a366"
server_key_iv: "6fc96a7947a9d323f90210235a40404c"
server_key_version: "encryptionkey@0"
I20260807 22:49:39.947192 3640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "a0d73308ec86426aaea9b1d375995e71"
format_stamp: "Formatted at 2026-08-07 22:49:39 on dist-test-slave-rgb1"
server_key: "448f64b818372f16fccedde0ecf8a366"
server_key_iv: "6fc96a7947a9d323f90210235a40404c"
server_key_version: "encryptionkey@0"
I20260807 22:49:39.948521 3640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:49:39.949545 3654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:39.949753 3640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:39.949841 3640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "a0d73308ec86426aaea9b1d375995e71"
format_stamp: "Formatted at 2026-08-07 22:49:39 on dist-test-slave-rgb1"
server_key: "448f64b818372f16fccedde0ecf8a366"
server_key_iv: "6fc96a7947a9d323f90210235a40404c"
server_key_version: "encryptionkey@0"
I20260807 22:49:39.949901 3640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:39.958024 3640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:39.958397 3640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:39.958559 3640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:39.958794 3640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:39.959136 3640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:39.959169 3640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:39.959193 3640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:39.959240 3640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:39.965117 3640 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:36195
I20260807 22:49:39.965215 3773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:36195 every 8 connection(s)
I20260807 22:49:39.965535 3640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:49:39.968468 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 3640
I20260807 22:49:39.968657 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:49:39.968864 16778 external_mini_cluster.cc:1428] Setting key 6ea54e92321d053cd6e4f7cac6d2894c
I20260807 22:49:39.969913 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43279
--builtin_ntp_servers=127.16.98.148:34247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 22:49:39.972009 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.966445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:54513 (local address 127.16.98.190:43279)
0807 22:49:39.966559 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:39.966562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:39.966783 (+ 221us) server_negotiation.cc:407] Connection header received
0807 22:49:39.967369 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:39.967370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:39.967383 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:39.967447 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:39.968149 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.968849 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.970262 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.970448 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.970538 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.971535 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:39.971536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:39.971536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:39.971540 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:39.971554 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:39.971558 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:39.971606 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:39.971838 (+ 232us) server_negotiation.cc:299] Negotiation successful
0807 22:49:39.971871 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 22:49:39.972285 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:39.966420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43279 (local address 127.16.98.130:54513)
0807 22:49:39.966641 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:39.966678 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:39.967247 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:39.967516 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:39.967520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:39.967528 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:39.968051 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.968055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.968959 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.968961 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.970125 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:39.970129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:39.970539 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:39.970541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:39.970712 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:39.971356 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:49:39.971356 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:39.971398 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:49:39.971428 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:39.971627 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:39.971635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:39.971638 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:49:39.971722 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 22:49:39.971963 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":110,"threads_started":1}
I20260807 22:49:39.972946 3774 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43279
I20260807 22:49:39.973066 3774 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:39.973240 3774 heartbeater.cc:511] Master 127.16.98.190:43279 requested a full tablet report, sending...
I20260807 22:49:39.973706 3443 ts_manager.cc:194] Registered new tserver with Master: a0d73308ec86426aaea9b1d375995e71 (127.16.98.130:36195)
I20260807 22:49:39.974282 3443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.98.130:54513
W20260807 22:49:40.062278 3777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:40.062506 3777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:40.062532 3777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:40.065510 3777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:40.065587 3777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:40.065629 3777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:49:40.067586 3777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:43279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:40.067884 3777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:40.068250 3777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:40.074189 3783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:40.074225 3782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:40.074219 3785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:40.074455 3777 server_base.cc:1034] running on GCE node
I20260807 22:49:40.074747 3777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:40.075006 3777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:40.076222 3777 hybrid_clock.cc:648] HybridClock initialized: now 1786142980076195 us; error 48 us; skew 500 ppm
I20260807 22:49:40.077787 3777 webserver.cc:460] Webserver started at http://127.16.98.131:45239/ using document root <none> and password file <none>
I20260807 22:49:40.078125 3777 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:40.078205 3777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:40.078372 3777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:40.079617 3777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "9db0289d8d634a6499ffba20f2347541"
format_stamp: "Formatted at 2026-08-07 22:49:40 on dist-test-slave-rgb1"
server_key: "6d894f6f52e45b00b7b77f0b390991b5"
server_key_iv: "a72fcb02ebccaee4935ee86f8caa466b"
server_key_version: "encryptionkey@0"
I20260807 22:49:40.080009 3777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "9db0289d8d634a6499ffba20f2347541"
format_stamp: "Formatted at 2026-08-07 22:49:40 on dist-test-slave-rgb1"
server_key: "6d894f6f52e45b00b7b77f0b390991b5"
server_key_iv: "a72fcb02ebccaee4935ee86f8caa466b"
server_key_version: "encryptionkey@0"
I20260807 22:49:40.082424 3777 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.004s
I20260807 22:49:40.083753 3791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:40.084201 3777 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:40.084409 3777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "9db0289d8d634a6499ffba20f2347541"
format_stamp: "Formatted at 2026-08-07 22:49:40 on dist-test-slave-rgb1"
server_key: "6d894f6f52e45b00b7b77f0b390991b5"
server_key_iv: "a72fcb02ebccaee4935ee86f8caa466b"
server_key_version: "encryptionkey@0"
I20260807 22:49:40.084546 3777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:40.101629 3777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:40.102133 3777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:40.102308 3777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:40.102586 3777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:40.102936 3777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:40.102993 3777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:40.103036 3777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:40.103066 3777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:40.111135 3777 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:41019
I20260807 22:49:40.111181 3908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:41019 every 8 connection(s)
I20260807 22:49:40.111552 3777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:49:40.117983 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 3777
I20260807 22:49:40.118113 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:49:40.118183 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:40.112420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:36057 (local address 127.16.98.190:43279)
0807 22:49:40.112534 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:40.112537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:40.112778 (+ 241us) server_negotiation.cc:407] Connection header received
0807 22:49:40.113385 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:40.113386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:40.113400 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:40.113466 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:40.114305 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.114886 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.116087 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.116309 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.116408 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:40.117540 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:40.117541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:40.117542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:40.117549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:40.117566 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:40.117571 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:40.117626 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:40.117934 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 22:49:40.118007 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 22:49:40.118316 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:40.112365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:43279 (local address 127.16.98.131:36057)
0807 22:49:40.112644 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:40.112682 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:40.113259 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:40.113572 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:40.113576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:40.113585 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:40.114198 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:40.114203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.115008 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.115009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:40.115989 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:40.115992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.116420 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.116422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:40.116678 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:40.117351 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:49:40.117352 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:40.117384 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:49:40.117409 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:40.117675 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:40.117684 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:40.117688 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:49:40.117829 (+ 141us) client_negotiation.cc:240] Negotiation successful
0807 22:49:40.118044 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":100,"threads_started":1}
I20260807 22:49:40.118340 16778 external_mini_cluster.cc:1428] Setting key 47a3654578ce712a9d9d55211323bb9f
I20260807 22:49:40.118866 3909 heartbeater.cc:347] Connected to a master server at 127.16.98.190:43279
I20260807 22:49:40.118996 3909 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:40.119204 3909 heartbeater.cc:511] Master 127.16.98.190:43279 requested a full tablet report, sending...
I20260807 22:49:40.119639 3443 ts_manager.cc:194] Registered new tserver with Master: 9db0289d8d634a6499ffba20f2347541 (127.16.98.131:41019)
I20260807 22:49:40.120083 3443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.98.131:36057
I20260807 22:49:40.130648 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:49:40.135735 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:40.132306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.16.98.190:43279)
0807 22:49:40.132398 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:40.132400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:40.132457 (+ 57us) server_negotiation.cc:407] Connection header received
0807 22:49:40.132486 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:40.132486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:40.132499 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:40.132649 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:40.132790 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.133316 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.134210 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.134409 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.134491 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:40.135369 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:40.135370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:40.135371 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:40.135375 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:40.135388 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:40.135422 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:40.135554 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:40.135640 (+ 86us) server_negotiation.cc:299] Negotiation successful
0807 22:49:40.135654 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 22:49:40.137900 3443 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:49:40.139700 3443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:49:40.146711 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:40.142774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.16.98.131:41019)
0807 22:49:40.142957 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:40.142962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:40.142971 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:49:40.143009 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:40.143010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:40.143025 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:40.143148 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:40.143378 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.144105 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.145593 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.146279 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.146371 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:40.146471 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:40.146571 (+ 100us) server_negotiation.cc:299] Negotiation successful
0807 22:49:40.146613 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":106,"threads_started":1}
I20260807 22:49:40.146704 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:40.142372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:41019 (local address 127.0.0.1:54048)
0807 22:49:40.142565 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:40.142606 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:40.142677 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:40.143244 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:40.143245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:40.143248 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:40.143294 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:40.143295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.144207 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.144208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:40.145500 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:40.145503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.146373 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.146374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:40.146464 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:40.146479 (+ 15us) client_negotiation.cc:769] Sending connection context
0807 22:49:40.146556 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 22:49:40.146614 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":103,"threads_started":1}
I20260807 22:49:40.146726 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:40.142375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:44625 (local address 127.0.0.1:42066)
0807 22:49:40.142775 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:40.142784 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:40.142822 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:40.143093 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:40.143096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:40.143099 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:40.143191 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:40.143193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.144534 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.144535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:40.145613 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:40.145615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.146447 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.146448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:40.146540 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:40.146545 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:49:40.146578 (+ 33us) client_negotiation.cc:240] Negotiation successful
0807 22:49:40.146621 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":106,"threads_started":1}
I20260807 22:49:40.147032 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:40.142602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.16.98.129:44625)
0807 22:49:40.142819 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:40.142825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:40.142836 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:49:40.142891 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:40.142893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:40.142910 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:40.143030 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:40.143289 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.144442 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.145675 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.146339 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.146769 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:40.146872 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:40.146901 (+ 29us) server_negotiation.cc:299] Negotiation successful
0807 22:49:40.146943 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
I20260807 22:49:40.147297 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:40.142759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.16.98.130:36195)
0807 22:49:40.142972 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:40.142976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:40.143286 (+ 310us) server_negotiation.cc:407] Connection header received
0807 22:49:40.143336 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:40.143339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:40.143353 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:40.143423 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:40.143635 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.144713 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.145906 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.146641 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.146703 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:40.146785 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:40.147187 (+ 402us) server_negotiation.cc:299] Negotiation successful
0807 22:49:40.147238 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":97,"threads_started":1}
I20260807 22:49:40.147478 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:40.142645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:36195 (local address 127.0.0.1:45140)
0807 22:49:40.143143 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:40.143156 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:40.143215 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:40.143514 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:40.143515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:40.143518 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:40.143553 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:40.143554 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.144803 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.144805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:40.145809 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:40.145813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.146918 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.146919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:40.146988 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:40.146993 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:49:40.147037 (+ 44us) client_negotiation.cc:240] Negotiation successful
0807 22:49:40.147085 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":58,"threads_started":1}
I20260807 22:49:40.147975 3570 tablet_service.cc:1467] Processing CreateTablet for tablet 38b5067c376940d28585c56d41268ae5 (DEFAULT_TABLE table=test-table [id=1e6ea08e8a124d74b4bc414be963b907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:40.147977 3841 tablet_service.cc:1467] Processing CreateTablet for tablet 38b5067c376940d28585c56d41268ae5 (DEFAULT_TABLE table=test-table [id=1e6ea08e8a124d74b4bc414be963b907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:40.148360 3841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38b5067c376940d28585c56d41268ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:40.148360 3570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38b5067c376940d28585c56d41268ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:40.149178 3705 tablet_service.cc:1467] Processing CreateTablet for tablet 38b5067c376940d28585c56d41268ae5 (DEFAULT_TABLE table=test-table [id=1e6ea08e8a124d74b4bc414be963b907]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:40.149528 3705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38b5067c376940d28585c56d41268ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:40.152087 3924 tablet_bootstrap.cc:492] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a: Bootstrap starting.
I20260807 22:49:40.152253 3923 tablet_bootstrap.cc:492] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541: Bootstrap starting.
I20260807 22:49:40.152626 3925 tablet_bootstrap.cc:492] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71: Bootstrap starting.
I20260807 22:49:40.152984 3923 tablet_bootstrap.cc:654] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:40.153311 3923 log.cc:824] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:40.153347 3925 tablet_bootstrap.cc:654] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:40.153759 3925 log.cc:824] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:40.154227 3924 tablet_bootstrap.cc:654] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:40.154435 3923 tablet_bootstrap.cc:492] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541: No bootstrap required, opened a new log
I20260807 22:49:40.154518 3923 ts_tablet_manager.cc:1397] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260807 22:49:40.154583 3924 log.cc:824] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:40.154608 3925 tablet_bootstrap.cc:492] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71: No bootstrap required, opened a new log
I20260807 22:49:40.154668 3925 ts_tablet_manager.cc:1397] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260807 22:49:40.155277 3924 tablet_bootstrap.cc:492] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a: No bootstrap required, opened a new log
I20260807 22:49:40.155361 3924 ts_tablet_manager.cc:1397] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260807 22:49:40.157292 3924 raft_consensus.cc:348] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1bd5b06a82496cb507d2394197c19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44625 } } peers { permanent_uuid: "a0d73308ec86426aaea9b1d375995e71" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36195 } } peers { permanent_uuid: "9db0289d8d634a6499ffba20f2347541" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41019 } }
I20260807 22:49:40.157459 3924 raft_consensus.cc:374] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:40.157506 3924 raft_consensus.cc:729] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1bd5b06a82496cb507d2394197c19a, State: Initialized, Role: FOLLOWER
I20260807 22:49:40.157527 3923 raft_consensus.cc:348] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1bd5b06a82496cb507d2394197c19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44625 } } peers { permanent_uuid: "a0d73308ec86426aaea9b1d375995e71" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36195 } } peers { permanent_uuid: "9db0289d8d634a6499ffba20f2347541" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41019 } }
I20260807 22:49:40.157565 3925 raft_consensus.cc:348] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1bd5b06a82496cb507d2394197c19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44625 } } peers { permanent_uuid: "a0d73308ec86426aaea9b1d375995e71" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36195 } } peers { permanent_uuid: "9db0289d8d634a6499ffba20f2347541" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41019 } }
I20260807 22:49:40.157653 3924 consensus_queue.cc:260] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1bd5b06a82496cb507d2394197c19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44625 } } peers { permanent_uuid: "a0d73308ec86426aaea9b1d375995e71" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36195 } } peers { permanent_uuid: "9db0289d8d634a6499ffba20f2347541" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41019 } }
I20260807 22:49:40.157711 3923 raft_consensus.cc:374] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:40.157723 3925 raft_consensus.cc:374] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:40.157765 3925 raft_consensus.cc:729] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d73308ec86426aaea9b1d375995e71, State: Initialized, Role: FOLLOWER
I20260807 22:49:40.157766 3923 raft_consensus.cc:729] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db0289d8d634a6499ffba20f2347541, State: Initialized, Role: FOLLOWER
I20260807 22:49:40.157899 3923 consensus_queue.cc:260] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1bd5b06a82496cb507d2394197c19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44625 } } peers { permanent_uuid: "a0d73308ec86426aaea9b1d375995e71" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36195 } } peers { permanent_uuid: "9db0289d8d634a6499ffba20f2347541" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41019 } }
I20260807 22:49:40.157917 3925 consensus_queue.cc:260] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1bd5b06a82496cb507d2394197c19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44625 } } peers { permanent_uuid: "a0d73308ec86426aaea9b1d375995e71" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36195 } } peers { permanent_uuid: "9db0289d8d634a6499ffba20f2347541" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41019 } }
I20260807 22:49:40.157974 3924 ts_tablet_manager.cc:1428] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:49:40.158128 3637 heartbeater.cc:503] Master 127.16.98.190:43279 was elected leader, sending a full tablet report...
I20260807 22:49:40.158187 3925 ts_tablet_manager.cc:1428] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260807 22:49:40.158187 3923 ts_tablet_manager.cc:1428] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260807 22:49:40.158315 3774 heartbeater.cc:503] Master 127.16.98.190:43279 was elected leader, sending a full tablet report...
I20260807 22:49:40.158320 3909 heartbeater.cc:503] Master 127.16.98.190:43279 was elected leader, sending a full tablet report...
I20260807 22:49:40.171633 3930 raft_consensus.cc:482] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:49:40.171780 3930 raft_consensus.cc:504] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1bd5b06a82496cb507d2394197c19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44625 } } peers { permanent_uuid: "a0d73308ec86426aaea9b1d375995e71" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36195 } } peers { permanent_uuid: "9db0289d8d634a6499ffba20f2347541" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41019 } }
I20260807 22:49:40.172211 3930 leader_election.cc:288] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c1bd5b06a82496cb507d2394197c19a (127.16.98.129:44625), 9db0289d8d634a6499ffba20f2347541 (127.16.98.131:41019)
I20260807 22:49:40.175222 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:40.172520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:44823 (local address 127.16.98.131:41019)
0807 22:49:40.172595 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:40.172598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:40.172747 (+ 149us) server_negotiation.cc:407] Connection header received
0807 22:49:40.172841 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:40.172841 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:40.172850 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:40.172916 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:40.173083 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.173562 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.174491 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.174848 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.174932 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:40.174941 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:40.175109 (+ 168us) server_negotiation.cc:299] Negotiation successful
0807 22:49:40.175148 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 22:49:40.175251 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:40.172474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:60727 (local address 127.16.98.129:44625)
0807 22:49:40.172533 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:40.172535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:40.172545 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:49:40.172607 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:40.172608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:40.172617 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:40.172683 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:40.172859 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.173379 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.174452 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.174859 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.174952 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:40.174961 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:40.175127 (+ 166us) server_negotiation.cc:299] Negotiation successful
0807 22:49:40.175171 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 22:49:40.175274 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:40.172390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:44625 (local address 127.16.98.130:60727)
0807 22:49:40.172455 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:40.172468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:40.172516 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:40.172690 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:40.172690 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:40.172692 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:40.172777 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:40.172779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.173483 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.173485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:40.174343 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:40.174346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.174950 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.174951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:40.175042 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:40.175045 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:49:40.175122 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 22:49:40.175167 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":3}
I20260807 22:49:40.175496 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:40.172473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:41019 (local address 127.16.98.130:44823)
0807 22:49:40.172675 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:40.172684 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:40.172762 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:40.172932 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:40.172933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:40.172935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:40.173007 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:40.173009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.173646 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.173648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:40.174399 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:40.174401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.174931 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.174932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:40.175019 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:40.175023 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:49:40.175099 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 22:49:40.175140 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":4,"thread_start_us":144,"threads_started":1}
I20260807 22:49:40.175676 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b5067c376940d28585c56d41268ae5" candidate_uuid: "a0d73308ec86426aaea9b1d375995e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1bd5b06a82496cb507d2394197c19a" is_pre_election: true
I20260807 22:49:40.175863 3590 raft_consensus.cc:2454] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d73308ec86426aaea9b1d375995e71 in term 0.
I20260807 22:49:40.175851 3861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b5067c376940d28585c56d41268ae5" candidate_uuid: "a0d73308ec86426aaea9b1d375995e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db0289d8d634a6499ffba20f2347541" is_pre_election: true
I20260807 22:49:40.176025 3861 raft_consensus.cc:2454] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d73308ec86426aaea9b1d375995e71 in term 0.
I20260807 22:49:40.176141 3655 leader_election.cc:302] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c1bd5b06a82496cb507d2394197c19a, a0d73308ec86426aaea9b1d375995e71; no voters:
I20260807 22:49:40.176282 3930 raft_consensus.cc:2781] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:49:40.176350 3930 raft_consensus.cc:482] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:49:40.176367 3930 raft_consensus.cc:3037] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:40.177192 3930 raft_consensus.cc:504] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1bd5b06a82496cb507d2394197c19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44625 } } peers { permanent_uuid: "a0d73308ec86426aaea9b1d375995e71" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36195 } } peers { permanent_uuid: "9db0289d8d634a6499ffba20f2347541" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41019 } }
I20260807 22:49:40.177345 3930 leader_election.cc:288] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [CANDIDATE]: Term 1 election: Requested vote from peers 5c1bd5b06a82496cb507d2394197c19a (127.16.98.129:44625), 9db0289d8d634a6499ffba20f2347541 (127.16.98.131:41019)
I20260807 22:49:40.177506 3861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b5067c376940d28585c56d41268ae5" candidate_uuid: "a0d73308ec86426aaea9b1d375995e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db0289d8d634a6499ffba20f2347541"
I20260807 22:49:40.177591 3861 raft_consensus.cc:3037] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:40.177592 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b5067c376940d28585c56d41268ae5" candidate_uuid: "a0d73308ec86426aaea9b1d375995e71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c1bd5b06a82496cb507d2394197c19a"
I20260807 22:49:40.177686 3590 raft_consensus.cc:3037] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:40.178368 3861 raft_consensus.cc:2454] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d73308ec86426aaea9b1d375995e71 in term 1.
I20260807 22:49:40.178545 3658 leader_election.cc:302] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9db0289d8d634a6499ffba20f2347541, a0d73308ec86426aaea9b1d375995e71; no voters:
I20260807 22:49:40.178566 3590 raft_consensus.cc:2454] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d73308ec86426aaea9b1d375995e71 in term 1.
I20260807 22:49:40.178651 3930 raft_consensus.cc:2781] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:40.178828 3930 raft_consensus.cc:686] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [term 1 LEADER]: Becoming Leader. State: Replica: a0d73308ec86426aaea9b1d375995e71, State: Running, Role: LEADER
I20260807 22:49:40.178936 3930 consensus_queue.cc:237] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1bd5b06a82496cb507d2394197c19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44625 } } peers { permanent_uuid: "a0d73308ec86426aaea9b1d375995e71" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36195 } } peers { permanent_uuid: "9db0289d8d634a6499ffba20f2347541" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41019 } }
I20260807 22:49:40.179780 3443 catalog_manager.cc:5495] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d73308ec86426aaea9b1d375995e71 (127.16.98.130). New cstate: current_term: 1 leader_uuid: "a0d73308ec86426aaea9b1d375995e71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1bd5b06a82496cb507d2394197c19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d73308ec86426aaea9b1d375995e71" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9db0289d8d634a6499ffba20f2347541" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41019 } health_report { overall_health: UNKNOWN } } }
I20260807 22:49:40.195595 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:40.192864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.16.98.130:36195)
0807 22:49:40.192938 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:40.192941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:40.192951 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:49:40.192997 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:40.192998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:40.193025 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:40.193145 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:49:40.193303 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.193839 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.194717 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:40.194934 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:40.195031 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:40.195191 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:49:40.195299 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:49:40.195424 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:40.195478 (+ 54us) server_negotiation.cc:299] Negotiation successful
0807 22:49:40.195491 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 22:49:40.197940 3861 raft_consensus.cc:1261] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541 [term 1 FOLLOWER]: Refusing update from remote peer a0d73308ec86426aaea9b1d375995e71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:49:40.197937 3590 raft_consensus.cc:1261] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a [term 1 FOLLOWER]: Refusing update from remote peer a0d73308ec86426aaea9b1d375995e71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:49:40.198326 3933 consensus_queue.cc:1035] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db0289d8d634a6499ffba20f2347541" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:49:40.198500 3930 consensus_queue.cc:1035] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c1bd5b06a82496cb507d2394197c19a" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:49:40.202332 3938 mvcc.cc:205] Tried to move back new op lower bound from 7316041646887084032 to 7316041646813630464. Current Snapshot: MvccSnapshot[applied={T|T < 7316041646887084032}]
I20260807 22:49:40.203851 3443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:49:40.203976 3443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:49:40.204846 3443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 277f524c51fc41c7bf102a116a7cb715: Sending DeleteTablet for 3 replicas of tablet 38b5067c376940d28585c56d41268ae5
I20260807 22:49:40.205281 3570 tablet_service.cc:1514] Processing DeleteTablet for tablet 38b5067c376940d28585c56d41268ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:40 UTC) from {username='slave'} at 127.0.0.1:42066
I20260807 22:49:40.205323 3841 tablet_service.cc:1514] Processing DeleteTablet for tablet 38b5067c376940d28585c56d41268ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:40 UTC) from {username='slave'} at 127.0.0.1:54048
I20260807 22:49:40.205601 3947 tablet_replica.cc:330] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a: stopping tablet replica
I20260807 22:49:40.205657 3705 tablet_service.cc:1514] Processing DeleteTablet for tablet 38b5067c376940d28585c56d41268ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:40 UTC) from {username='slave'} at 127.0.0.1:45140
I20260807 22:49:40.205724 3947 raft_consensus.cc:2229] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:49:40.205849 3947 raft_consensus.cc:2258] T 38b5067c376940d28585c56d41268ae5 P 5c1bd5b06a82496cb507d2394197c19a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:49:40.205976 3948 tablet_replica.cc:330] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71: stopping tablet replica
I20260807 22:49:40.206079 3948 raft_consensus.cc:2229] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:49:40.206205 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 3505
I20260807 22:49:40.206226 3948 raft_consensus.cc:2258] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:49:40.210592 3948 ts_tablet_manager.cc:1905] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:49:40.213032 3948 ts_tablet_manager.cc:1918] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:49:40.213135 3948 log.cc:1196] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/38b5067c376940d28585c56d41268ae5
I20260807 22:49:40.213447 3948 ts_tablet_manager.cc:1939] T 38b5067c376940d28585c56d41268ae5 P a0d73308ec86426aaea9b1d375995e71: Deleting consensus metadata
I20260807 22:49:40.213675 3946 tablet_replica.cc:330] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541: stopping tablet replica
I20260807 22:49:40.213836 3946 raft_consensus.cc:2229] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:49:40.213984 3946 raft_consensus.cc:2258] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:49:40.214275 3426 catalog_manager.cc:4841] TS a0d73308ec86426aaea9b1d375995e71 (127.16.98.130:36195): tablet 38b5067c376940d28585c56d41268ae5 (table test-table [id=1e6ea08e8a124d74b4bc414be963b907]) successfully deleted
I20260807 22:49:40.214550 3946 ts_tablet_manager.cc:1905] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 22:49:40.216670 3426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:44625 (error 108)
I20260807 22:49:40.216728 3946 ts_tablet_manager.cc:1918] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:49:40.216796 3946 log.cc:1196] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/38b5067c376940d28585c56d41268ae5
I20260807 22:49:40.217106 3946 ts_tablet_manager.cc:1939] T 38b5067c376940d28585c56d41268ae5 P 9db0289d8d634a6499ffba20f2347541: Deleting consensus metadata
I20260807 22:49:40.217643 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:40.217475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:44625 (local address 127.0.0.1:42070)
0807 22:49:40.217547 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:40.217599 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:44625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 22:49:40.217646 3429 catalog_manager.cc:4841] TS 9db0289d8d634a6499ffba20f2347541 (127.16.98.131:41019): tablet 38b5067c376940d28585c56d41268ae5 (table test-table [id=1e6ea08e8a124d74b4bc414be963b907]) successfully deleted
I20260807 22:49:40.217751 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 3640
W20260807 22:49:40.217787 3426 catalog_manager.cc:4584] TS 5c1bd5b06a82496cb507d2394197c19a (127.16.98.129:44625): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38b5067c376940d28585c56d41268ae5: Network error: Client connection negotiation failed: client connection to 127.16.98.129:44625: connect: Connection refused (error 111)
I20260807 22:49:40.225153 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 3777
I20260807 22:49:40.231353 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 3410
2026-08-07T22:49:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (771 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:49:40 dist-test-slave-rgb1 krb5kdc[3952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:49:40 dist-test-slave-rgb1 krb5kdc[3952](info): set up 2 sockets
Aug 07 22:49:40 dist-test-slave-rgb1 krb5kdc[3952](info): commencing operation
krb5kdc: starting...
W20260807 22:49:41.340708 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:49:41 dist-test-slave-rgb1 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:49:41.489295 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:40205
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:40205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:41.577888 3966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:41.578135 3966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:41.578161 3966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:41.579312 3966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:41.579349 3966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:41.579365 3966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:41.579377 3966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:41.579389 3966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:41.580899 3966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:40205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:40205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:41.581123 3966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:41.581314 3966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:41.586542 3971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:41.586567 3972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:41.586705 3966 server_base.cc:1034] running on GCE node
W20260807 22:49:41.586594 3974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:41.587080 3966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:41.587399 3966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:41.588634 3966 hybrid_clock.cc:648] HybridClock initialized: now 1786142981588602 us; error 44 us; skew 500 ppm
Aug 07 22:49:41 dist-test-slave-rgb1 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142981, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:41.591609 3966 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:49:41.592162 3966 webserver.cc:460] Webserver started at http://127.16.98.190:32857/ using document root <none> and password file <none>
I20260807 22:49:41.592434 3966 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:41.592509 3966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:41.592741 3966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:41.593901 3966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "5b948310dbd64e2593979e68b6b927e1"
format_stamp: "Formatted at 2026-08-07 22:49:41 on dist-test-slave-rgb1"
server_key: "1fe8183979a29c318eca54997d1fff08"
server_key_iv: "331094ae090c9a67f29d0974433e812b"
server_key_version: "encryptionkey@0"
I20260807 22:49:41.594285 3966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "5b948310dbd64e2593979e68b6b927e1"
format_stamp: "Formatted at 2026-08-07 22:49:41 on dist-test-slave-rgb1"
server_key: "1fe8183979a29c318eca54997d1fff08"
server_key_iv: "331094ae090c9a67f29d0974433e812b"
server_key_version: "encryptionkey@0"
I20260807 22:49:41.595891 3966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:49:41.596992 3981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:41.597265 3966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:41.597380 3966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "5b948310dbd64e2593979e68b6b927e1"
format_stamp: "Formatted at 2026-08-07 22:49:41 on dist-test-slave-rgb1"
server_key: "1fe8183979a29c318eca54997d1fff08"
server_key_iv: "331094ae090c9a67f29d0974433e812b"
server_key_version: "encryptionkey@0"
I20260807 22:49:41.597463 3966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:41.621810 3966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:41.624302 3966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:41.624512 3966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:41.629406 3966 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:40205
I20260807 22:49:41.629468 4037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:40205 every 8 connection(s)
I20260807 22:49:41.629876 3966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:41.630918 4038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:41.633481 4038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1: Bootstrap starting.
I20260807 22:49:41.634095 4038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:41.634409 4038 log.cc:824] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:41.635345 4038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1: No bootstrap required, opened a new log
I20260807 22:49:41.636130 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 3966
I20260807 22:49:41.636281 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:41.636586 16778 external_mini_cluster.cc:1428] Setting key 35c232135388b61ba4e07eb35735d522
I20260807 22:49:41.637992 4038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b948310dbd64e2593979e68b6b927e1" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40205 } }
I20260807 22:49:41.638177 4038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:41.638229 4038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b948310dbd64e2593979e68b6b927e1, State: Initialized, Role: FOLLOWER
I20260807 22:49:41.638381 4038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b948310dbd64e2593979e68b6b927e1" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40205 } }
I20260807 22:49:41.638480 4038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:41.638537 4038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:41.638617 4038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:41.639631 4038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b948310dbd64e2593979e68b6b927e1" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40205 } }
I20260807 22:49:41.639766 4038 leader_election.cc:302] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b948310dbd64e2593979e68b6b927e1; no voters:
I20260807 22:49:41.639967 4038 leader_election.cc:288] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:41.640015 4044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:41.640161 4044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [term 1 LEADER]: Becoming Leader. State: Replica: 5b948310dbd64e2593979e68b6b927e1, State: Running, Role: LEADER
I20260807 22:49:41.640256 4044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b948310dbd64e2593979e68b6b927e1" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40205 } }
I20260807 22:49:41.640293 4038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:41.640657 4046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b948310dbd64e2593979e68b6b927e1. Latest consensus state: current_term: 1 leader_uuid: "5b948310dbd64e2593979e68b6b927e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b948310dbd64e2593979e68b6b927e1" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40205 } } }
I20260807 22:49:41.640740 4046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:41.641361 4049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:41.641893 4049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:41.642115 4045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b948310dbd64e2593979e68b6b927e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b948310dbd64e2593979e68b6b927e1" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40205 } } }
I20260807 22:49:41.642328 4045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:41.643787 4049 catalog_manager.cc:1347] Generated new cluster ID: fa730d3e305d4977b62a80549c0b89fb
I20260807 22:49:41.643852 4049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 22:49:41 dist-test-slave-rgb1 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:49:41.649611 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:41.637293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.16.98.190:40205)
0807 22:49:41.637475 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:41.637483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:41.637509 (+ 26us) server_negotiation.cc:407] Connection header received
0807 22:49:41.637993 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:41.638001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:41.638055 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:41.638256 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:41.639120 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:41.640224 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:41.641674 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:41.642294 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:41.642410 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:41.645639 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:41.645643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:41.645644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:41.645657 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:41.647698 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:41.648106 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:41.648110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:41.648110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:41.648249 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:41.648469 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:41.648470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:41.648470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:41.648834 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:41.649030 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:41.649103 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 22:49:41.649389 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260807 22:49:41.659307 4049 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:41.660193 4049 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:41.668370 4049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1: Generated new TSK 0
I20260807 22:49:41.668689 4049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.98.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
Entry for principal kudu/127.16.98.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab.
I20260807 22:49:41.708887 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:40205
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:41.798537 4066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:41.798795 4066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:41.798842 4066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:41.800161 4066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:41.800236 4066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:41.800298 4066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:49:41.802093 4066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.129.keytab
--principal=kudu/127.16.98.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:40205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:41.802379 4066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:41.802644 4066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:41.807909 4072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:41.808075 4074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:41.808059 4066 server_base.cc:1034] running on GCE node
W20260807 22:49:41.807904 4071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:41.808414 4066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:41.808740 4066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:41.809959 4066 hybrid_clock.cc:648] HybridClock initialized: now 1786142981809920 us; error 51 us; skew 500 ppm
Aug 07 22:49:41 dist-test-slave-rgb1 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142981, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:41.813190 4066 init.cc:377] Logged in from keytab as kudu/127.16.98.129@KRBTEST.COM (short username kudu)
I20260807 22:49:41.813707 4066 webserver.cc:460] Webserver started at http://127.16.98.129:36501/ using document root <none> and password file <none>
I20260807 22:49:41.813954 4066 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:41.814009 4066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:41.814118 4066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:41.815393 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "a90017153c7144a6988c87ecca0e1c3e"
format_stamp: "Formatted at 2026-08-07 22:49:41 on dist-test-slave-rgb1"
server_key: "b869204c8994aa48ccfa773f376bf785"
server_key_iv: "630c9ee275385e4dd432b3f22fd48e02"
server_key_version: "encryptionkey@0"
I20260807 22:49:41.815749 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "a90017153c7144a6988c87ecca0e1c3e"
format_stamp: "Formatted at 2026-08-07 22:49:41 on dist-test-slave-rgb1"
server_key: "b869204c8994aa48ccfa773f376bf785"
server_key_iv: "630c9ee275385e4dd432b3f22fd48e02"
server_key_version: "encryptionkey@0"
I20260807 22:49:41.817405 4066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:49:41.818231 4082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:41.818480 4066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:41.818574 4066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "a90017153c7144a6988c87ecca0e1c3e"
format_stamp: "Formatted at 2026-08-07 22:49:41 on dist-test-slave-rgb1"
server_key: "b869204c8994aa48ccfa773f376bf785"
server_key_iv: "630c9ee275385e4dd432b3f22fd48e02"
server_key_version: "encryptionkey@0"
I20260807 22:49:41.818642 4066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:41.833923 4066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:41.836707 4066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:41.836851 4066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:41.837085 4066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:41.837458 4066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:41.837497 4066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:41.837532 4066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:41.837570 4066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:41.844945 4066 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:37483
I20260807 22:49:41.845045 4199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:37483 every 8 connection(s)
I20260807 22:49:41.845412 4066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:49:41.845849 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 4066
I20260807 22:49:41.846011 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:49:41.846254 16778 external_mini_cluster.cc:1428] Setting key 92430a66a3be8062e6d05d151d41ddaf
Aug 07 22:49:41 dist-test-slave-rgb1 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142981, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.129@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.98.130@KRBTEST.COM; defaulting to no policy
I20260807 22:49:41.858458 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:41.846416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:34181 (local address 127.16.98.190:40205)
0807 22:49:41.846472 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:41.846475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:41.846822 (+ 347us) server_negotiation.cc:407] Connection header received
0807 22:49:41.847550 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:41.847551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:41.847571 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:41.847612 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:41.849329 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:41.850234 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:41.851413 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:41.851835 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:41.851913 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:41.854567 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:41.854570 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:41.854571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:41.854596 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:41.856780 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:41.857298 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:41.857300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:41.857301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:41.857363 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:41.857678 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:41.857679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:41.857679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:41.857828 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:41.857908 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:41.858291 (+ 383us) server_negotiation.cc:299] Negotiation successful
0807 22:49:41.858386 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260807 22:49:41.858664 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:41.846336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:40205 (local address 127.16.98.129:34181)
0807 22:49:41.846607 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:41.846666 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:41.847391 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:41.847728 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:41.847732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:41.848818 (+ 1086us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:41.849165 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:41.849173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:41.850371 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:41.850372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:41.851307 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:41.851310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:41.851946 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:41.851947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:41.852142 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:41.852858 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:49:41.852867 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:41.854399 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:41.856883 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:41.856891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:41.856893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:41.857203 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:41.857469 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:41.857470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:41.857470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:41.857593 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:41.857906 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:41.857972 (+ 66us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:41.858142 (+ 170us) client_negotiation.cc:769] Sending connection context
0807 22:49:41.858242 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 22:49:41.858397 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":129,"threads_started":1}
I20260807 22:49:41.859298 4200 heartbeater.cc:347] Connected to a master server at 127.16.98.190:40205
I20260807 22:49:41.859391 4200 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:41.859661 4200 heartbeater.cc:511] Master 127.16.98.190:40205 requested a full tablet report, sending...
I20260807 22:49:41.860270 3999 ts_manager.cc:194] Registered new tserver with Master: a90017153c7144a6988c87ecca0e1c3e (127.16.98.129:37483)
I20260807 22:49:41.861317 3999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.129@KRBTEST.COM'} at 127.16.98.129:34181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
Entry for principal kudu/127.16.98.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab.
I20260807 22:49:41.881263 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:40205
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:41.972596 4205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:41.972846 4205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:41.972889 4205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:41.974378 4205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:41.974458 4205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:41.974494 4205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:49:41.976253 4205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.130.keytab
--principal=kudu/127.16.98.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:40205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:41.976512 4205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:41.976837 4205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:41.983265 4214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:41.983433 4205 server_base.cc:1034] running on GCE node
W20260807 22:49:41.983407 4211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:41.983407 4212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:41.984011 4205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:41.984278 4205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:41.985443 4205 hybrid_clock.cc:648] HybridClock initialized: now 1786142981985407 us; error 67 us; skew 500 ppm
Aug 07 22:49:41 dist-test-slave-rgb1 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142981, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:41.988498 4205 init.cc:377] Logged in from keytab as kudu/127.16.98.130@KRBTEST.COM (short username kudu)
I20260807 22:49:41.989033 4205 webserver.cc:460] Webserver started at http://127.16.98.130:40623/ using document root <none> and password file <none>
I20260807 22:49:41.989288 4205 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:41.989334 4205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:41.989508 4205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:41.990658 4205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "3981ac4954ba4463b681d4ce97ba1ca5"
format_stamp: "Formatted at 2026-08-07 22:49:41 on dist-test-slave-rgb1"
server_key: "83ec59d24b028c0600471dfbf0f3fa59"
server_key_iv: "79f66c4bea24c7e4e81ed823531e7f41"
server_key_version: "encryptionkey@0"
I20260807 22:49:41.991045 4205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "3981ac4954ba4463b681d4ce97ba1ca5"
format_stamp: "Formatted at 2026-08-07 22:49:41 on dist-test-slave-rgb1"
server_key: "83ec59d24b028c0600471dfbf0f3fa59"
server_key_iv: "79f66c4bea24c7e4e81ed823531e7f41"
server_key_version: "encryptionkey@0"
I20260807 22:49:41.992480 4205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260807 22:49:41.993685 4222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:41.994129 4205 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:49:41.994460 4205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "3981ac4954ba4463b681d4ce97ba1ca5"
format_stamp: "Formatted at 2026-08-07 22:49:41 on dist-test-slave-rgb1"
server_key: "83ec59d24b028c0600471dfbf0f3fa59"
server_key_iv: "79f66c4bea24c7e4e81ed823531e7f41"
server_key_version: "encryptionkey@0"
I20260807 22:49:41.994597 4205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:42.010136 4205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:42.014487 4205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:42.014712 4205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:42.015018 4205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:42.015395 4205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:42.015456 4205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:42.015497 4205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:42.015548 4205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:42.023880 4205 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:36403
I20260807 22:49:42.023973 4339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:36403 every 8 connection(s)
I20260807 22:49:42.024390 4205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:49:42.028756 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 4205
I20260807 22:49:42.028905 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:49:42.029160 16778 external_mini_cluster.cc:1428] Setting key a9c673f86128a62c2a6d37d1dad9d073
Aug 07 22:49:42 dist-test-slave-rgb1 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142981, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.130@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:49:42.035372 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.025333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:52083 (local address 127.16.98.190:40205)
0807 22:49:42.025409 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:42.025411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:42.025655 (+ 244us) server_negotiation.cc:407] Connection header received
0807 22:49:42.026183 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:42.026184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:42.026196 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:42.026266 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:42.027393 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.027910 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.029506 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.029773 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.029846 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.032275 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:42.032276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:42.032276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:42.032285 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:42.033861 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:42.034327 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:42.034328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:42.034329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:42.034391 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:42.034669 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:42.034670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:42.034670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:42.034796 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:42.034875 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:42.035167 (+ 292us) server_negotiation.cc:299] Negotiation successful
0807 22:49:42.035266 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 22:49:42.035597 4342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.025290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:40205 (local address 127.16.98.130:52083)
0807 22:49:42.025481 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:42.025531 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:42.026064 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:42.026346 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:42.026349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:42.027046 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:42.027263 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.027269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.028215 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.028216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.029401 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.029403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.029884 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.029885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.030059 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.030652 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:49:42.030660 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:42.032164 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:42.033972 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:42.033982 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:42.033984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:42.034244 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:42.034463 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:42.034464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:42.034464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:42.034593 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:42.034866 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:42.034874 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:42.035024 (+ 150us) client_negotiation.cc:769] Sending connection context
0807 22:49:42.035146 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 22:49:42.035335 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260807 22:49:42.036145 4340 heartbeater.cc:347] Connected to a master server at 127.16.98.190:40205
I20260807 22:49:42.036235 4340 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:42.036453 4340 heartbeater.cc:511] Master 127.16.98.190:40205 requested a full tablet report, sending...
I20260807 22:49:42.036969 3999 ts_manager.cc:194] Registered new tserver with Master: 3981ac4954ba4463b681d4ce97ba1ca5 (127.16.98.130:36403)
I20260807 22:49:42.037533 3999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.130@KRBTEST.COM'} at 127.16.98.130:52083
WARNING: no policy specified for kudu/127.16.98.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
Entry for principal kudu/127.16.98.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab.
I20260807 22:49:42.064931 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:40205
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:42.156883 4345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:42.157145 4345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:42.157174 4345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:42.158432 4345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:42.158483 4345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:42.158514 4345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:49:42.160434 4345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.131.keytab
--principal=kudu/127.16.98.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:40205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:42.160826 4345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:42.161325 4345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:42.167886 4351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:42.167915 4350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:42.168007 4354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:42.167883 4345 server_base.cc:1034] running on GCE node
I20260807 22:49:42.168723 4345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:42.169054 4345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:42.170259 4345 hybrid_clock.cc:648] HybridClock initialized: now 1786142982170228 us; error 46 us; skew 500 ppm
Aug 07 22:49:42 dist-test-slave-rgb1 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142982, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:42.174614 4345 init.cc:377] Logged in from keytab as kudu/127.16.98.131@KRBTEST.COM (short username kudu)
I20260807 22:49:42.175144 4345 webserver.cc:460] Webserver started at http://127.16.98.131:42965/ using document root <none> and password file <none>
I20260807 22:49:42.175562 4345 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:42.175782 4345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:42.175966 4345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:42.177385 4345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "5ed041f6dffa473482384bbf8d67cb44"
format_stamp: "Formatted at 2026-08-07 22:49:42 on dist-test-slave-rgb1"
server_key: "4a097f0f8d1b8f29413ddd8df424d5cd"
server_key_iv: "3f2cd11a1c5ec7e7c6594f4061bfb29d"
server_key_version: "encryptionkey@0"
I20260807 22:49:42.177798 4345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "5ed041f6dffa473482384bbf8d67cb44"
format_stamp: "Formatted at 2026-08-07 22:49:42 on dist-test-slave-rgb1"
server_key: "4a097f0f8d1b8f29413ddd8df424d5cd"
server_key_iv: "3f2cd11a1c5ec7e7c6594f4061bfb29d"
server_key_version: "encryptionkey@0"
I20260807 22:49:42.180161 4345 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 22:49:42.182006 4361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:42.182456 4345 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:49:42.182727 4345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "5ed041f6dffa473482384bbf8d67cb44"
format_stamp: "Formatted at 2026-08-07 22:49:42 on dist-test-slave-rgb1"
server_key: "4a097f0f8d1b8f29413ddd8df424d5cd"
server_key_iv: "3f2cd11a1c5ec7e7c6594f4061bfb29d"
server_key_version: "encryptionkey@0"
I20260807 22:49:42.182837 4345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:42.203991 4345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:42.206589 4345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:42.206736 4345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:42.207079 4345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:42.207418 4345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:42.207455 4345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:42.207479 4345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:42.207491 4345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:42.213616 4345 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:42897
I20260807 22:49:42.213738 4479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:42897 every 8 connection(s)
I20260807 22:49:42.214054 4345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
Aug 07 22:49:42 dist-test-slave-rgb1 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142982, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.131@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:49:42.222193 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 4345
I20260807 22:49:42.222339 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:49:42.222705 16778 external_mini_cluster.cc:1428] Setting key 60235525a731a5036b17f7a7de0effe7
I20260807 22:49:42.226464 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.214963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:60757 (local address 127.16.98.190:40205)
0807 22:49:42.215059 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:42.215062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:42.215250 (+ 188us) server_negotiation.cc:407] Connection header received
0807 22:49:42.215785 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:42.215786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:42.215798 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:42.215873 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:42.217225 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.217839 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.219190 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.219384 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.219495 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.222646 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:42.222648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:42.222649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:42.222660 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:42.224500 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:42.225105 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:42.225106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:42.225107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:42.225168 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:42.225571 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:42.225573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:42.225574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:42.225880 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:42.225953 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:42.226315 (+ 362us) server_negotiation.cc:299] Negotiation successful
0807 22:49:42.226392 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 22:49:42.226693 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.214870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:40205 (local address 127.16.98.131:60757)
0807 22:49:42.215096 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:42.215142 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:42.215668 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:42.215959 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:42.215963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:42.216736 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:42.216960 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.216966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.217999 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.218000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.219071 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.219075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.219511 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.219512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.219630 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.220241 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:49:42.220250 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:42.222498 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:42.224666 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:42.224674 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:42.224676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:42.224937 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:42.225266 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:49:42.225267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:49:42.225268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:49:42.225408 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:49:42.225970 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:42.225977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:42.226139 (+ 162us) client_negotiation.cc:769] Sending connection context
0807 22:49:42.226247 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 22:49:42.226397 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":106,"threads_started":1}
I20260807 22:49:42.227463 4480 heartbeater.cc:347] Connected to a master server at 127.16.98.190:40205
I20260807 22:49:42.227582 4480 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:42.227746 4480 heartbeater.cc:511] Master 127.16.98.190:40205 requested a full tablet report, sending...
I20260807 22:49:42.228206 3999 ts_manager.cc:194] Registered new tserver with Master: 5ed041f6dffa473482384bbf8d67cb44 (127.16.98.131:42897)
I20260807 22:49:42.228837 3999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.98.131@KRBTEST.COM'} at 127.16.98.131:60757
I20260807 22:49:42.235988 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:49:42.246157 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.238303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.16.98.190:40205)
0807 22:49:42.238433 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:42.238435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:42.238572 (+ 137us) server_negotiation.cc:407] Connection header received
0807 22:49:42.238686 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:42.238687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:42.238701 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:42.238786 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:42.239460 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.240037 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.241047 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.241258 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.241339 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.242773 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:42.242775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:42.242775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:42.242784 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:42.244652 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:42.245044 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:42.245059 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:42.245060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:42.245127 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:42.245353 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:42.245354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:42.245354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:42.245530 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:42.245648 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:42.245789 (+ 141us) server_negotiation.cc:299] Negotiation successful
0807 22:49:42.245979 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 22:49:42.248256 3999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:49:42.250046 3999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:49:42.258047 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.252899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:42897 (local address 127.0.0.1:49668)
0807 22:49:42.253226 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:42.253235 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:42.253278 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:42.253704 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:42.253708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:42.253711 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:42.253820 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.253823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.255264 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.255265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.256769 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.256773 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.257706 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.257707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.257776 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.257796 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 22:49:42.257862 (+ 66us) client_negotiation.cc:240] Negotiation successful
0807 22:49:42.257928 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":207,"thread_start_us":70,"threads_started":1}
I20260807 22:49:42.258354 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.252902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:36403 (local address 127.0.0.1:51966)
0807 22:49:42.253378 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:42.253387 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:42.253423 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:42.253772 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:42.253773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:42.253775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:42.253827 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.253829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.256282 (+ 2453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.256283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.257236 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.257239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.258126 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.258127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.258199 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.258204 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:49:42.258255 (+ 51us) client_negotiation.cc:240] Negotiation successful
0807 22:49:42.258291 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260807 22:49:42.258433 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.253231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.16.98.131:42897)
0807 22:49:42.253406 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:42.253411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:42.253424 (+ 13us) server_negotiation.cc:407] Connection header received
0807 22:49:42.253552 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:42.253554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:42.253570 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:42.253640 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:42.253910 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.255130 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.256881 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.257541 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.257613 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.257671 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:42.258309 (+ 638us) server_negotiation.cc:299] Negotiation successful
0807 22:49:42.258353 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
I20260807 22:49:42.258425 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.253062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.16.98.130:36403)
0807 22:49:42.253552 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:42.253556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:42.253567 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:49:42.253605 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:42.253608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:42.253624 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:42.253696 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:42.255266 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.256174 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.257338 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.258051 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.258114 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.258173 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:42.258291 (+ 118us) server_negotiation.cc:299] Negotiation successful
0807 22:49:42.258339 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":86,"threads_started":1}
I20260807 22:49:42.258728 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.253176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39596 (local address 127.16.98.129:37483)
0807 22:49:42.253418 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:42.253434 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:42.253443 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:49:42.253489 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:42.253491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:42.253509 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:42.253605 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:42.253963 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.255130 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.256866 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.257541 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.257629 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.257686 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:42.257988 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 22:49:42.258035 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":102,"threads_started":1}
I20260807 22:49:42.258759 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.252899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:37483 (local address 127.0.0.1:39596)
0807 22:49:42.253157 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:42.253201 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:42.253277 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:42.253704 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:42.253708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:42.253713 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:42.253820 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.253823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.255263 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.255264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.256771 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.256773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.257630 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.257631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.257774 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.257794 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 22:49:42.257869 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 22:49:42.257935 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":9,"thread_start_us":83,"threads_started":1}
I20260807 22:49:42.259191 4271 tablet_service.cc:1467] Processing CreateTablet for tablet 443cb58e5f6e4cc18cb252a262a1bf27 (DEFAULT_TABLE table=test-table [id=47ce46f4763245d0bed318f3c4856683]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:42.259439 4133 tablet_service.cc:1467] Processing CreateTablet for tablet 443cb58e5f6e4cc18cb252a262a1bf27 (DEFAULT_TABLE table=test-table [id=47ce46f4763245d0bed318f3c4856683]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:42.259542 4271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 443cb58e5f6e4cc18cb252a262a1bf27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:42.259742 4133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 443cb58e5f6e4cc18cb252a262a1bf27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:42.261010 4412 tablet_service.cc:1467] Processing CreateTablet for tablet 443cb58e5f6e4cc18cb252a262a1bf27 (DEFAULT_TABLE table=test-table [id=47ce46f4763245d0bed318f3c4856683]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:42.261345 4412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 443cb58e5f6e4cc18cb252a262a1bf27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:42.262542 4499 tablet_bootstrap.cc:492] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5: Bootstrap starting.
I20260807 22:49:42.262814 4500 tablet_bootstrap.cc:492] T 443cb58e5f6e4cc18cb252a262a1bf27 P a90017153c7144a6988c87ecca0e1c3e: Bootstrap starting.
I20260807 22:49:42.263532 4499 tablet_bootstrap.cc:654] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:42.263532 4500 tablet_bootstrap.cc:654] T 443cb58e5f6e4cc18cb252a262a1bf27 P a90017153c7144a6988c87ecca0e1c3e: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:42.264210 4500 log.cc:824] T 443cb58e5f6e4cc18cb252a262a1bf27 P a90017153c7144a6988c87ecca0e1c3e: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:42.264385 4501 tablet_bootstrap.cc:492] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44: Bootstrap starting.
I20260807 22:49:42.264621 4499 log.cc:824] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:42.265136 4501 tablet_bootstrap.cc:654] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:42.265373 4499 tablet_bootstrap.cc:492] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5: No bootstrap required, opened a new log
I20260807 22:49:42.265373 4500 tablet_bootstrap.cc:492] T 443cb58e5f6e4cc18cb252a262a1bf27 P a90017153c7144a6988c87ecca0e1c3e: No bootstrap required, opened a new log
I20260807 22:49:42.265450 4500 ts_tablet_manager.cc:1397] T 443cb58e5f6e4cc18cb252a262a1bf27 P a90017153c7144a6988c87ecca0e1c3e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 22:49:42.265450 4499 ts_tablet_manager.cc:1397] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260807 22:49:42.265733 4501 log.cc:824] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:42.266426 4501 tablet_bootstrap.cc:492] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44: No bootstrap required, opened a new log
I20260807 22:49:42.266510 4501 ts_tablet_manager.cc:1397] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 22:49:42.267470 4499 raft_consensus.cc:348] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a90017153c7144a6988c87ecca0e1c3e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37483 } } peers { permanent_uuid: "3981ac4954ba4463b681d4ce97ba1ca5" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36403 } } peers { permanent_uuid: "5ed041f6dffa473482384bbf8d67cb44" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42897 } }
I20260807 22:49:42.267470 4500 raft_consensus.cc:348] T 443cb58e5f6e4cc18cb252a262a1bf27 P a90017153c7144a6988c87ecca0e1c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a90017153c7144a6988c87ecca0e1c3e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37483 } } peers { permanent_uuid: "3981ac4954ba4463b681d4ce97ba1ca5" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36403 } } peers { permanent_uuid: "5ed041f6dffa473482384bbf8d67cb44" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42897 } }
I20260807 22:49:42.267663 4499 raft_consensus.cc:374] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:42.267663 4500 raft_consensus.cc:374] T 443cb58e5f6e4cc18cb252a262a1bf27 P a90017153c7144a6988c87ecca0e1c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:42.267704 4499 raft_consensus.cc:729] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3981ac4954ba4463b681d4ce97ba1ca5, State: Initialized, Role: FOLLOWER
I20260807 22:49:42.267704 4500 raft_consensus.cc:729] T 443cb58e5f6e4cc18cb252a262a1bf27 P a90017153c7144a6988c87ecca0e1c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90017153c7144a6988c87ecca0e1c3e, State: Initialized, Role: FOLLOWER
I20260807 22:49:42.267827 4500 consensus_queue.cc:260] T 443cb58e5f6e4cc18cb252a262a1bf27 P a90017153c7144a6988c87ecca0e1c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a90017153c7144a6988c87ecca0e1c3e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37483 } } peers { permanent_uuid: "3981ac4954ba4463b681d4ce97ba1ca5" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36403 } } peers { permanent_uuid: "5ed041f6dffa473482384bbf8d67cb44" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42897 } }
I20260807 22:49:42.267840 4499 consensus_queue.cc:260] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a90017153c7144a6988c87ecca0e1c3e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37483 } } peers { permanent_uuid: "3981ac4954ba4463b681d4ce97ba1ca5" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36403 } } peers { permanent_uuid: "5ed041f6dffa473482384bbf8d67cb44" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42897 } }
I20260807 22:49:42.268126 4500 ts_tablet_manager.cc:1428] T 443cb58e5f6e4cc18cb252a262a1bf27 P a90017153c7144a6988c87ecca0e1c3e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:49:42.268126 4499 ts_tablet_manager.cc:1428] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260807 22:49:42.268172 4200 heartbeater.cc:503] Master 127.16.98.190:40205 was elected leader, sending a full tablet report...
I20260807 22:49:42.268277 4340 heartbeater.cc:503] Master 127.16.98.190:40205 was elected leader, sending a full tablet report...
I20260807 22:49:42.268082 4501 raft_consensus.cc:348] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a90017153c7144a6988c87ecca0e1c3e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37483 } } peers { permanent_uuid: "3981ac4954ba4463b681d4ce97ba1ca5" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36403 } } peers { permanent_uuid: "5ed041f6dffa473482384bbf8d67cb44" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42897 } }
I20260807 22:49:42.268656 4501 raft_consensus.cc:374] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:42.268708 4501 raft_consensus.cc:729] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed041f6dffa473482384bbf8d67cb44, State: Initialized, Role: FOLLOWER
I20260807 22:49:42.268824 4501 consensus_queue.cc:260] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a90017153c7144a6988c87ecca0e1c3e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37483 } } peers { permanent_uuid: "3981ac4954ba4463b681d4ce97ba1ca5" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36403 } } peers { permanent_uuid: "5ed041f6dffa473482384bbf8d67cb44" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42897 } }
I20260807 22:49:42.269131 4501 ts_tablet_manager.cc:1428] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:49:42.269162 4480 heartbeater.cc:503] Master 127.16.98.190:40205 was elected leader, sending a full tablet report...
W20260807 22:49:42.275087 4341 tablet.cc:2369] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 22:49:42.346328 4201 tablet.cc:2369] T 443cb58e5f6e4cc18cb252a262a1bf27 P a90017153c7144a6988c87ecca0e1c3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:49:42.382339 4506 raft_consensus.cc:482] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:49:42.382489 4506 raft_consensus.cc:504] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a90017153c7144a6988c87ecca0e1c3e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37483 } } peers { permanent_uuid: "3981ac4954ba4463b681d4ce97ba1ca5" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36403 } } peers { permanent_uuid: "5ed041f6dffa473482384bbf8d67cb44" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42897 } }
I20260807 22:49:42.382858 4506 leader_election.cc:288] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a90017153c7144a6988c87ecca0e1c3e (127.16.98.129:37483), 5ed041f6dffa473482384bbf8d67cb44 (127.16.98.131:42897)
I20260807 22:49:42.386466 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.383211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:43541 (local address 127.16.98.129:37483)
0807 22:49:42.383261 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:42.383263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:42.383491 (+ 228us) server_negotiation.cc:407] Connection header received
0807 22:49:42.383573 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:42.383574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:42.383589 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:42.383684 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:42.383865 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.384449 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.385560 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.385988 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.386084 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.386094 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:42.386314 (+ 220us) server_negotiation.cc:299] Negotiation successful
0807 22:49:42.386358 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 22:49:42.386463 4342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.383115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:42897 (local address 127.16.98.130:52515)
0807 22:49:42.383214 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:42.383230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:42.383303 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:42.383629 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:42.383637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:42.383639 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:42.383718 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.383720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.384433 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.384434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.385420 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.385422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.386114 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.386115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.386210 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.386213 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:49:42.386312 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 22:49:42.386359 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":1}
I20260807 22:49:42.386559 4509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.383115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:37483 (local address 127.16.98.130:43541)
0807 22:49:42.383377 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:42.383387 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:42.383507 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:42.383686 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:42.383687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:42.383689 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:42.383776 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.383778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.384553 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.384554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.385448 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.385451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.386082 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.386083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.386172 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.386177 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:49:42.386294 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 22:49:42.386342 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":53,"thread_start_us":142,"threads_started":1}
I20260807 22:49:42.386641 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.383272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:52515 (local address 127.16.98.131:42897)
0807 22:49:42.383327 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:42.383329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:42.383340 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:49:42.383374 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:42.383375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:42.383385 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:42.383453 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:42.383805 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.384338 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.385524 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.386007 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.386115 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.386125 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:42.386514 (+ 389us) server_negotiation.cc:299] Negotiation successful
0807 22:49:42.386561 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 22:49:42.386940 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443cb58e5f6e4cc18cb252a262a1bf27" candidate_uuid: "3981ac4954ba4463b681d4ce97ba1ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed041f6dffa473482384bbf8d67cb44" is_pre_election: true
I20260807 22:49:42.387080 4434 raft_consensus.cc:2454] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3981ac4954ba4463b681d4ce97ba1ca5 in term 0.
I20260807 22:49:42.387320 4225 leader_election.cc:302] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3981ac4954ba4463b681d4ce97ba1ca5, 5ed041f6dffa473482384bbf8d67cb44; no voters:
I20260807 22:49:42.387307 4153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443cb58e5f6e4cc18cb252a262a1bf27" candidate_uuid: "3981ac4954ba4463b681d4ce97ba1ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90017153c7144a6988c87ecca0e1c3e" is_pre_election: true
I20260807 22:49:42.387454 4153 raft_consensus.cc:2454] T 443cb58e5f6e4cc18cb252a262a1bf27 P a90017153c7144a6988c87ecca0e1c3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3981ac4954ba4463b681d4ce97ba1ca5 in term 0.
I20260807 22:49:42.387522 4506 raft_consensus.cc:2781] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:49:42.387595 4506 raft_consensus.cc:482] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:49:42.387629 4506 raft_consensus.cc:3037] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:42.388440 4506 raft_consensus.cc:504] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a90017153c7144a6988c87ecca0e1c3e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37483 } } peers { permanent_uuid: "3981ac4954ba4463b681d4ce97ba1ca5" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36403 } } peers { permanent_uuid: "5ed041f6dffa473482384bbf8d67cb44" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42897 } }
I20260807 22:49:42.388649 4506 leader_election.cc:288] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [CANDIDATE]: Term 1 election: Requested vote from peers a90017153c7144a6988c87ecca0e1c3e (127.16.98.129:37483), 5ed041f6dffa473482384bbf8d67cb44 (127.16.98.131:42897)
I20260807 22:49:42.388845 4153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443cb58e5f6e4cc18cb252a262a1bf27" candidate_uuid: "3981ac4954ba4463b681d4ce97ba1ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90017153c7144a6988c87ecca0e1c3e"
I20260807 22:49:42.388947 4153 raft_consensus.cc:3037] T 443cb58e5f6e4cc18cb252a262a1bf27 P a90017153c7144a6988c87ecca0e1c3e [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:42.388916 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443cb58e5f6e4cc18cb252a262a1bf27" candidate_uuid: "3981ac4954ba4463b681d4ce97ba1ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed041f6dffa473482384bbf8d67cb44"
I20260807 22:49:42.389048 4434 raft_consensus.cc:3037] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:42.389818 4153 raft_consensus.cc:2454] T 443cb58e5f6e4cc18cb252a262a1bf27 P a90017153c7144a6988c87ecca0e1c3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3981ac4954ba4463b681d4ce97ba1ca5 in term 1.
I20260807 22:49:42.390023 4434 raft_consensus.cc:2454] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3981ac4954ba4463b681d4ce97ba1ca5 in term 1.
I20260807 22:49:42.390040 4224 leader_election.cc:302] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3981ac4954ba4463b681d4ce97ba1ca5, a90017153c7144a6988c87ecca0e1c3e; no voters:
I20260807 22:49:42.390246 4506 raft_consensus.cc:2781] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:42.390413 4506 raft_consensus.cc:686] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [term 1 LEADER]: Becoming Leader. State: Replica: 3981ac4954ba4463b681d4ce97ba1ca5, State: Running, Role: LEADER
I20260807 22:49:42.390558 4506 consensus_queue.cc:237] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a90017153c7144a6988c87ecca0e1c3e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37483 } } peers { permanent_uuid: "3981ac4954ba4463b681d4ce97ba1ca5" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36403 } } peers { permanent_uuid: "5ed041f6dffa473482384bbf8d67cb44" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42897 } }
I20260807 22:49:42.391556 3998 catalog_manager.cc:5495] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3981ac4954ba4463b681d4ce97ba1ca5 (127.16.98.130). New cstate: current_term: 1 leader_uuid: "3981ac4954ba4463b681d4ce97ba1ca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a90017153c7144a6988c87ecca0e1c3e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3981ac4954ba4463b681d4ce97ba1ca5" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 36403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ed041f6dffa473482384bbf8d67cb44" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42897 } health_report { overall_health: UNKNOWN } } }
I20260807 22:49:42.405742 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.402878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.16.98.130:36403)
0807 22:49:42.402939 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:42.402941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:42.402952 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:49:42.403199 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:42.403200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:42.403227 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:42.403314 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:49:42.403510 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.403997 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.404836 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.405057 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.405156 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.405349 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:49:42.405477 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:49:42.405579 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:42.405598 (+ 19us) server_negotiation.cc:299] Negotiation successful
0807 22:49:42.405623 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260807 22:49:42.408109 4434 raft_consensus.cc:1261] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44 [term 1 FOLLOWER]: Refusing update from remote peer 3981ac4954ba4463b681d4ce97ba1ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:49:42.408180 4153 raft_consensus.cc:1261] T 443cb58e5f6e4cc18cb252a262a1bf27 P a90017153c7144a6988c87ecca0e1c3e [term 1 FOLLOWER]: Refusing update from remote peer 3981ac4954ba4463b681d4ce97ba1ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:49:42.408481 4506 consensus_queue.cc:1035] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed041f6dffa473482384bbf8d67cb44" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:49:42.408723 4510 consensus_queue.cc:1035] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a90017153c7144a6988c87ecca0e1c3e" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 37483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:49:42.411392 4515 mvcc.cc:205] Tried to move back new op lower bound from 7316041655939010560 to 7316041655872344064. Current Snapshot: MvccSnapshot[applied={T|T < 7316041655939010560}]
I20260807 22:49:42.412130 4514 mvcc.cc:205] Tried to move back new op lower bound from 7316041655939010560 to 7316041655872344064. Current Snapshot: MvccSnapshot[applied={T|T < 7316041655939010560}]
I20260807 22:49:42.415102 3999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:49:42.415243 3999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:49:42.416250 3999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b948310dbd64e2593979e68b6b927e1: Sending DeleteTablet for 3 replicas of tablet 443cb58e5f6e4cc18cb252a262a1bf27
I20260807 22:49:42.416725 4133 tablet_service.cc:1514] Processing DeleteTablet for tablet 443cb58e5f6e4cc18cb252a262a1bf27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:42 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:39596
I20260807 22:49:42.416704 4412 tablet_service.cc:1514] Processing DeleteTablet for tablet 443cb58e5f6e4cc18cb252a262a1bf27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:42 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:49668
I20260807 22:49:42.417146 4271 tablet_service.cc:1514] Processing DeleteTablet for tablet 443cb58e5f6e4cc18cb252a262a1bf27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:42 UTC) from {username='kudu', principal='kudu/127.16.98.190@KRBTEST.COM'} at 127.0.0.1:51966
I20260807 22:49:42.417304 4524 tablet_replica.cc:330] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44: stopping tablet replica
I20260807 22:49:42.417446 4524 raft_consensus.cc:2229] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:49:42.417599 4524 raft_consensus.cc:2258] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:49:42.417613 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 4066
I20260807 22:49:42.418267 4525 tablet_replica.cc:330] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5: stopping tablet replica
I20260807 22:49:42.418941 4525 raft_consensus.cc:2229] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:49:42.419102 4525 raft_consensus.cc:2258] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:49:42.419648 4525 ts_tablet_manager.cc:1905] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:49:42.419661 4524 ts_tablet_manager.cc:1905] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:49:42.422135 4524 ts_tablet_manager.cc:1918] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:49:42.422209 4525 ts_tablet_manager.cc:1918] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:49:42.422302 4524 log.cc:1196] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/443cb58e5f6e4cc18cb252a262a1bf27
I20260807 22:49:42.422309 4525 log.cc:1196] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/443cb58e5f6e4cc18cb252a262a1bf27
I20260807 22:49:42.422636 4524 ts_tablet_manager.cc:1939] T 443cb58e5f6e4cc18cb252a262a1bf27 P 5ed041f6dffa473482384bbf8d67cb44: Deleting consensus metadata
I20260807 22:49:42.422648 4525 ts_tablet_manager.cc:1939] T 443cb58e5f6e4cc18cb252a262a1bf27 P 3981ac4954ba4463b681d4ce97ba1ca5: Deleting consensus metadata
I20260807 22:49:42.423158 3985 catalog_manager.cc:4841] TS 3981ac4954ba4463b681d4ce97ba1ca5 (127.16.98.130:36403): tablet 443cb58e5f6e4cc18cb252a262a1bf27 (table test-table [id=47ce46f4763245d0bed318f3c4856683]) successfully deleted
I20260807 22:49:42.423224 3984 catalog_manager.cc:4841] TS 5ed041f6dffa473482384bbf8d67cb44 (127.16.98.131:42897): tablet 443cb58e5f6e4cc18cb252a262a1bf27 (table test-table [id=47ce46f4763245d0bed318f3c4856683]) successfully deleted
W20260807 22:49:42.428642 3983 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:37483 (error 108)
I20260807 22:49:42.429025 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 4205
I20260807 22:49:42.429788 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.429250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:37483 (local address 127.0.0.1:39598)
0807 22:49:42.429694 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:42.429738 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419}
W20260807 22:49:42.430730 3983 catalog_manager.cc:4584] TS a90017153c7144a6988c87ecca0e1c3e (127.16.98.129:37483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 443cb58e5f6e4cc18cb252a262a1bf27: Network error: Client connection negotiation failed: client connection to 127.16.98.129:37483: connect: Connection refused (error 111)
I20260807 22:49:42.436228 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 4345
I20260807 22:49:42.442958 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 3966
2026-08-07T22:49:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2212 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-07T22:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:42Z Disabled control of system clock
I20260807 22:49:42.475580 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:37323
--webserver_interface=127.16.98.190
--webserver_port=0
--builtin_ntp_servers=127.16.98.148:46735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:37323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 22:49:42.566592 4530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:42.566843 4530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:42.566866 4530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:42.568081 4530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:42.568130 4530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:42.568143 4530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:42.568156 4530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:42.568168 4530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:42.569657 4530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:37323
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:37323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:42.569941 4530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:42.570272 4530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:42.575812 4538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:42.575867 4535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:42.575922 4536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:42.575946 4530 server_base.cc:1034] running on GCE node
I20260807 22:49:42.576252 4530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:42.576694 4530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:42.577895 4530 hybrid_clock.cc:648] HybridClock initialized: now 1786142982577887 us; error 42 us; skew 500 ppm
I20260807 22:49:42.579551 4530 webserver.cc:460] Webserver started at http://127.16.98.190:45831/ using document root <none> and password file <none>
I20260807 22:49:42.579878 4530 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:42.579950 4530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:42.580116 4530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:42.581966 4530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "2a3078c3d9e3499d85e7794f7441a62d"
format_stamp: "Formatted at 2026-08-07 22:49:42 on dist-test-slave-rgb1"
server_key: "7eb53b739342297e15a762f0d8b3d6fd"
server_key_iv: "0e53fc34c46fab87e07c5fbc83757709"
server_key_version: "encryptionkey@0"
I20260807 22:49:42.582398 4530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "2a3078c3d9e3499d85e7794f7441a62d"
format_stamp: "Formatted at 2026-08-07 22:49:42 on dist-test-slave-rgb1"
server_key: "7eb53b739342297e15a762f0d8b3d6fd"
server_key_iv: "0e53fc34c46fab87e07c5fbc83757709"
server_key_version: "encryptionkey@0"
I20260807 22:49:42.584029 4530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260807 22:49:42.585062 4544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:42.585302 4530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 22:49:42.585429 4530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "2a3078c3d9e3499d85e7794f7441a62d"
format_stamp: "Formatted at 2026-08-07 22:49:42 on dist-test-slave-rgb1"
server_key: "7eb53b739342297e15a762f0d8b3d6fd"
server_key_iv: "0e53fc34c46fab87e07c5fbc83757709"
server_key_version: "encryptionkey@0"
I20260807 22:49:42.585543 4530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:42.615713 4530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:42.616127 4530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:42.616312 4530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:42.620983 4530 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:37323
I20260807 22:49:42.621141 4598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:37323 every 8 connection(s)
I20260807 22:49:42.621500 4530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:42.622538 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 4530
I20260807 22:49:42.622670 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:42.622843 4599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:42.622928 16778 external_mini_cluster.cc:1428] Setting key 549f1159b96803543f8d48daf299fcd7
I20260807 22:49:42.626606 4599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d: Bootstrap starting.
I20260807 22:49:42.627279 4599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:42.627604 4599 log.cc:824] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:42.628672 4599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d: No bootstrap required, opened a new log
I20260807 22:49:42.630054 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.623515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.16.98.190:37323)
0807 22:49:42.623747 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:42.623757 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:42.623795 (+ 38us) server_negotiation.cc:407] Connection header received
0807 22:49:42.624422 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:42.624432 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:42.624530 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:42.624730 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:42.624923 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.626297 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.627398 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.628049 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.628181 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.629236 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:42.629240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:42.629240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:42.629423 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:42.629458 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:42.629470 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:42.629578 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:42.629679 (+ 101us) server_negotiation.cc:299] Negotiation successful
0807 22:49:42.629808 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260807 22:49:42.630998 4599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3078c3d9e3499d85e7794f7441a62d" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37323 } }
I20260807 22:49:42.631165 4599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:42.631217 4599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3078c3d9e3499d85e7794f7441a62d, State: Initialized, Role: FOLLOWER
I20260807 22:49:42.631376 4599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3078c3d9e3499d85e7794f7441a62d" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37323 } }
I20260807 22:49:42.631459 4599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:42.631510 4599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:42.631561 4599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:42.632157 4599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3078c3d9e3499d85e7794f7441a62d" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37323 } }
I20260807 22:49:42.632277 4599 leader_election.cc:302] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a3078c3d9e3499d85e7794f7441a62d; no voters:
I20260807 22:49:42.632499 4599 leader_election.cc:288] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:42.632722 4605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:42.632809 4599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:42.632903 4605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [term 1 LEADER]: Becoming Leader. State: Replica: 2a3078c3d9e3499d85e7794f7441a62d, State: Running, Role: LEADER
I20260807 22:49:42.632998 4605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3078c3d9e3499d85e7794f7441a62d" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37323 } }
I20260807 22:49:42.633323 4605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a3078c3d9e3499d85e7794f7441a62d. Latest consensus state: current_term: 1 leader_uuid: "2a3078c3d9e3499d85e7794f7441a62d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3078c3d9e3499d85e7794f7441a62d" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37323 } } }
I20260807 22:49:42.633415 4605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:42.633345 4604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a3078c3d9e3499d85e7794f7441a62d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3078c3d9e3499d85e7794f7441a62d" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 37323 } } }
I20260807 22:49:42.633610 4604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:42.633626 4614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:42.634208 4614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:42.635897 4614 catalog_manager.cc:1347] Generated new cluster ID: e91616ba7ac542f5be16240fa962f5d9
I20260807 22:49:42.636022 4614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:49:42.649289 4614 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:42.649932 4614 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:42.657449 4614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d: Generated new TSK 0
I20260807 22:49:42.657707 4614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:49:42.663252 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.129:0
--local_ip_for_outbound_sockets=127.16.98.129
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37323
--builtin_ntp_servers=127.16.98.148:46735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 22:49:42.764819 4625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:42.765069 4625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:42.765096 4625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:42.766628 4625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:42.766685 4625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:42.766716 4625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.129
I20260807 22:49:42.768345 4625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.98.129
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.129
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:42.768699 4625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:42.769088 4625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 22:49:42.775400 4625 server_base.cc:1034] running on GCE node
W20260807 22:49:42.775359 4633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:42.775450 4630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:42.775363 4631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:42.775904 4625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:42.776201 4625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:42.777421 4625 hybrid_clock.cc:648] HybridClock initialized: now 1786142982777391 us; error 45 us; skew 500 ppm
I20260807 22:49:42.778949 4625 webserver.cc:460] Webserver started at http://127.16.98.129:41073/ using document root <none> and password file <none>
I20260807 22:49:42.779268 4625 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:42.779364 4625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:42.779534 4625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:42.780933 4625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/data/instance:
uuid: "64655259b3d74217b63f8257dd92c39f"
format_stamp: "Formatted at 2026-08-07 22:49:42 on dist-test-slave-rgb1"
server_key: "a77c433372d93aaefb97218e620d3262"
server_key_iv: "f08b0f24716fce649801741aeff2fc05"
server_key_version: "encryptionkey@0"
I20260807 22:49:42.781329 4625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance:
uuid: "64655259b3d74217b63f8257dd92c39f"
format_stamp: "Formatted at 2026-08-07 22:49:42 on dist-test-slave-rgb1"
server_key: "a77c433372d93aaefb97218e620d3262"
server_key_iv: "f08b0f24716fce649801741aeff2fc05"
server_key_version: "encryptionkey@0"
I20260807 22:49:42.782959 4625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:49:42.783895 4639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:42.784061 4625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:42.784160 4625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/wal
uuid: "64655259b3d74217b63f8257dd92c39f"
format_stamp: "Formatted at 2026-08-07 22:49:42 on dist-test-slave-rgb1"
server_key: "a77c433372d93aaefb97218e620d3262"
server_key_iv: "f08b0f24716fce649801741aeff2fc05"
server_key_version: "encryptionkey@0"
I20260807 22:49:42.784246 4625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:42.794726 4625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:42.795091 4625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:42.795262 4625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:42.795521 4625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:42.795887 4625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:42.795924 4625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:42.795950 4625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:42.796001 4625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:42.803138 4625 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.129:44555
I20260807 22:49:42.803231 4753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.129:44555 every 8 connection(s)
I20260807 22:49:42.803598 4625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/data/info.pb
I20260807 22:49:42.809788 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.804534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.129:46635 (local address 127.16.98.190:37323)
0807 22:49:42.804631 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:42.804634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:42.804818 (+ 184us) server_negotiation.cc:407] Connection header received
0807 22:49:42.805399 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:42.805401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:42.805417 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:42.805536 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:42.806436 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.806954 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.808107 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.808296 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.808367 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.809457 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:42.809459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:42.809459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:42.809465 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:42.809481 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:42.809485 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:42.809557 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:42.809656 (+ 99us) server_negotiation.cc:299] Negotiation successful
0807 22:49:42.809699 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 22:49:42.810042 4756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.804454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:37323 (local address 127.16.98.129:46635)
0807 22:49:42.804682 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:42.804720 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:42.805265 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:42.805607 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:42.805612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:42.805622 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:42.806311 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.806316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.807059 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.807060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.808009 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.808012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.808380 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.808381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.808499 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.809278 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:49:42.809278 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:42.809309 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:49:42.809334 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:42.809559 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:42.809567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:42.809571 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:49:42.809661 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 22:49:42.809769 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260807 22:49:42.810412 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 4625
I20260807 22:49:42.810541 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-0/wal/instance
I20260807 22:49:42.810638 4754 heartbeater.cc:347] Connected to a master server at 127.16.98.190:37323
I20260807 22:49:42.810748 4754 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:42.810802 16778 external_mini_cluster.cc:1428] Setting key 8d56691958f31084d1bd0ba448271848
I20260807 22:49:42.810968 4754 heartbeater.cc:511] Master 127.16.98.190:37323 requested a full tablet report, sending...
I20260807 22:49:42.811666 4561 ts_manager.cc:194] Registered new tserver with Master: 64655259b3d74217b63f8257dd92c39f (127.16.98.129:44555)
I20260807 22:49:42.812068 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.130:0
--local_ip_for_outbound_sockets=127.16.98.130
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37323
--builtin_ntp_servers=127.16.98.148:46735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 22:49:42.812493 4561 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.98.129:46635
W20260807 22:49:42.901122 4757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:42.901351 4757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:42.901376 4757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:42.902519 4757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:42.902558 4757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:42.902585 4757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.130
I20260807 22:49:42.903976 4757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.98.130
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.130
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:42.904217 4757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:42.904502 4757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:42.910065 4766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:42.910100 4763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:42.910100 4764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:42.910145 4757 server_base.cc:1034] running on GCE node
I20260807 22:49:42.910528 4757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:42.910843 4757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:42.912040 4757 hybrid_clock.cc:648] HybridClock initialized: now 1786142982912067 us; error 80 us; skew 500 ppm
I20260807 22:49:42.913743 4757 webserver.cc:460] Webserver started at http://127.16.98.130:42435/ using document root <none> and password file <none>
I20260807 22:49:42.914253 4757 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:42.914322 4757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:42.914605 4757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:42.916042 4757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/data/instance:
uuid: "e7b91165e1d94f3ab55c120348f34d7f"
format_stamp: "Formatted at 2026-08-07 22:49:42 on dist-test-slave-rgb1"
server_key: "f8a99c27797895856f22c102ebe62b16"
server_key_iv: "62cdcc3010b258d60642fd4c9524697f"
server_key_version: "encryptionkey@0"
I20260807 22:49:42.916447 4757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance:
uuid: "e7b91165e1d94f3ab55c120348f34d7f"
format_stamp: "Formatted at 2026-08-07 22:49:42 on dist-test-slave-rgb1"
server_key: "f8a99c27797895856f22c102ebe62b16"
server_key_iv: "62cdcc3010b258d60642fd4c9524697f"
server_key_version: "encryptionkey@0"
I20260807 22:49:42.918253 4757 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260807 22:49:42.920526 4772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:42.921000 4757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 22:49:42.921260 4757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/wal
uuid: "e7b91165e1d94f3ab55c120348f34d7f"
format_stamp: "Formatted at 2026-08-07 22:49:42 on dist-test-slave-rgb1"
server_key: "f8a99c27797895856f22c102ebe62b16"
server_key_iv: "62cdcc3010b258d60642fd4c9524697f"
server_key_version: "encryptionkey@0"
I20260807 22:49:42.921535 4757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:42.936769 4757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:42.937142 4757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:42.937314 4757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:42.937599 4757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:42.937983 4757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:42.938021 4757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:42.938045 4757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:42.938058 4757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:42.946055 4757 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.130:38181
I20260807 22:49:42.946154 4886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.130:38181 every 8 connection(s)
I20260807 22:49:42.946619 4757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/data/info.pb
I20260807 22:49:42.948871 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 4757
I20260807 22:49:42.949015 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/wal/instance
I20260807 22:49:42.949256 16778 external_mini_cluster.cc:1428] Setting key d283b60d5352bfaf4508eb28c1cc013c
I20260807 22:49:42.950676 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.98.131:0
--local_ip_for_outbound_sockets=127.16.98.131
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37323
--builtin_ntp_servers=127.16.98.148:46735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 22:49:42.953970 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.947585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:60393 (local address 127.16.98.190:37323)
0807 22:49:42.947678 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:42.947681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:42.948025 (+ 344us) server_negotiation.cc:407] Connection header received
0807 22:49:42.948705 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:42.948706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:42.948718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:42.948794 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:42.949875 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.950579 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.952163 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.952364 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.952468 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.953625 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:42.953627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:42.953627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:42.953634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:42.953661 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:42.953665 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:42.953743 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:42.953851 (+ 108us) server_negotiation.cc:299] Negotiation successful
0807 22:49:42.953884 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 22:49:42.954217 4889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:42.947561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:37323 (local address 127.16.98.130:60393)
0807 22:49:42.947848 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:42.947907 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:42.948485 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:42.948914 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:42.948920 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:42.948932 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:42.949775 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.949781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.950715 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.950716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.952013 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:42.952017 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:42.952485 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:42.952486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:42.952683 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:42.953365 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:49:42.953366 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:42.953399 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:49:42.953424 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:42.953749 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:42.953757 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:42.953760 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:49:42.953855 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 22:49:42.953968 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":126,"threads_started":1}
I20260807 22:49:42.954996 4887 heartbeater.cc:347] Connected to a master server at 127.16.98.190:37323
I20260807 22:49:42.955109 4887 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:42.955433 4887 heartbeater.cc:511] Master 127.16.98.190:37323 requested a full tablet report, sending...
I20260807 22:49:42.956022 4561 ts_manager.cc:194] Registered new tserver with Master: e7b91165e1d94f3ab55c120348f34d7f (127.16.98.130:38181)
I20260807 22:49:42.956643 4561 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.98.130:60393
W20260807 22:49:43.041483 4890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:43.041714 4890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:43.041738 4890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:43.042927 4890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:43.042973 4890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:43.042999 4890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.98.131
I20260807 22:49:43.044319 4890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.98.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.98.131
--webserver_port=0
--tserver_master_addrs=127.16.98.190:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.98.131
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:43.044518 4890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:43.044862 4890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:43.050076 4896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:43.050084 4895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:43.050171 4898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:43.050535 4890 server_base.cc:1034] running on GCE node
I20260807 22:49:43.050751 4890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:43.051002 4890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:43.052163 4890 hybrid_clock.cc:648] HybridClock initialized: now 1786142983052154 us; error 33 us; skew 500 ppm
I20260807 22:49:43.053537 4890 webserver.cc:460] Webserver started at http://127.16.98.131:41911/ using document root <none> and password file <none>
I20260807 22:49:43.053817 4890 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:43.053891 4890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:43.054052 4890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:43.055141 4890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/data/instance:
uuid: "5c8e75e542d64598a5ba65cf151ad1fb"
format_stamp: "Formatted at 2026-08-07 22:49:43 on dist-test-slave-rgb1"
server_key: "01288c86d3c0073039ebcb86de47d48a"
server_key_iv: "d720b0f843cc47e616d6c33f7e6f04f9"
server_key_version: "encryptionkey@0"
I20260807 22:49:43.055501 4890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance:
uuid: "5c8e75e542d64598a5ba65cf151ad1fb"
format_stamp: "Formatted at 2026-08-07 22:49:43 on dist-test-slave-rgb1"
server_key: "01288c86d3c0073039ebcb86de47d48a"
server_key_iv: "d720b0f843cc47e616d6c33f7e6f04f9"
server_key_version: "encryptionkey@0"
I20260807 22:49:43.056937 4890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:49:43.057827 4904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:43.058012 4890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:43.058163 4890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/wal
uuid: "5c8e75e542d64598a5ba65cf151ad1fb"
format_stamp: "Formatted at 2026-08-07 22:49:43 on dist-test-slave-rgb1"
server_key: "01288c86d3c0073039ebcb86de47d48a"
server_key_iv: "d720b0f843cc47e616d6c33f7e6f04f9"
server_key_version: "encryptionkey@0"
I20260807 22:49:43.058221 4890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:43.068472 4890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:43.068825 4890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:43.068940 4890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:43.069131 4890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 22:49:43.069482 4890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:49:43.069518 4890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:43.069564 4890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:49:43.069592 4890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:43.075980 4890 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.131:34955
I20260807 22:49:43.076031 5019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.131:34955 every 8 connection(s)
I20260807 22:49:43.076375 4890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/data/info.pb
I20260807 22:49:43.078256 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 4890
I20260807 22:49:43.078387 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/wal/instance
I20260807 22:49:43.078608 16778 external_mini_cluster.cc:1428] Setting key 2b02a6acf9ea2d1a13c1e1acf46dfea0
I20260807 22:49:43.082283 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:43.077216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.131:39743 (local address 127.16.98.190:37323)
0807 22:49:43.077293 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:43.077296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:43.077566 (+ 270us) server_negotiation.cc:407] Connection header received
0807 22:49:43.078074 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:43.078076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:43.078091 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:43.078174 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:43.078968 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.079685 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.080674 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.080895 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.080973 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:43.081961 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:43.081962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:43.081963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:43.081968 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:43.081985 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:43.081989 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:43.082049 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:43.082196 (+ 147us) server_negotiation.cc:299] Negotiation successful
0807 22:49:43.082220 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 22:49:43.082433 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:43.077217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:37323 (local address 127.16.98.131:39743)
0807 22:49:43.077436 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:43.077476 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:43.077970 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:43.078221 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:43.078224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:43.078232 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:49:43.078850 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:43.078855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.079760 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.079761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:43.080599 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:43.080602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.080996 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.080997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:43.081140 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:43.081777 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 22:49:43.081778 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:49:43.081807 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 22:49:43.081831 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:49:43.082089 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:49:43.082095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:49:43.082097 (+ 2us) client_negotiation.cc:769] Sending connection context
0807 22:49:43.082154 (+ 57us) client_negotiation.cc:240] Negotiation successful
0807 22:49:43.082237 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260807 22:49:43.082965 5020 heartbeater.cc:347] Connected to a master server at 127.16.98.190:37323
I20260807 22:49:43.083067 5020 heartbeater.cc:464] Registering TS with master...
I20260807 22:49:43.083263 5020 heartbeater.cc:511] Master 127.16.98.190:37323 requested a full tablet report, sending...
I20260807 22:49:43.083706 4561 ts_manager.cc:194] Registered new tserver with Master: 5c8e75e542d64598a5ba65cf151ad1fb (127.16.98.131:34955)
I20260807 22:49:43.084041 4561 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.98.131:39743
I20260807 22:49:43.090855 16778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:49:43.095880 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:43.092627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.16.98.190:37323)
0807 22:49:43.092697 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:43.092699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:43.092892 (+ 193us) server_negotiation.cc:407] Connection header received
0807 22:49:43.092924 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:43.092925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:43.092936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:43.093004 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:43.093260 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.093782 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.094606 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.094800 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.094857 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:43.095615 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:43.095616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:43.095616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:43.095621 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:43.095634 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:43.095638 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:43.095690 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:43.095798 (+ 108us) server_negotiation.cc:299] Negotiation successful
0807 22:49:43.095819 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260807 22:49:43.097923 4561 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 22:49:43.099758 4561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:49:43.107484 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:43.102647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.16.98.130:38181)
0807 22:49:43.102894 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:43.102899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:43.102912 (+ 13us) server_negotiation.cc:407] Connection header received
0807 22:49:43.102960 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:43.102963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:43.102980 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:43.103140 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:43.103712 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.104937 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.106520 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.107037 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.107139 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:43.107241 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:43.107334 (+ 93us) server_negotiation.cc:299] Negotiation successful
0807 22:49:43.107378 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":116,"threads_started":1}
I20260807 22:49:43.107465 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:43.102459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.130:38181 (local address 127.0.0.1:60652)
0807 22:49:43.102633 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:43.102675 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:43.102746 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:43.103152 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:43.103155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:43.103158 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:43.103330 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:43.103335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.105035 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.105036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:43.106410 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:43.106414 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.107144 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.107145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:43.107226 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:43.107240 (+ 14us) client_negotiation.cc:769] Sending connection context
0807 22:49:43.107318 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 22:49:43.107378 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":192,"thread_start_us":62,"threads_started":1}
I20260807 22:49:43.107678 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:43.102459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:34955 (local address 127.0.0.1:50712)
0807 22:49:43.103262 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:43.103275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:43.103339 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:43.103647 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:43.103648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:43.103651 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:43.103702 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:43.103704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.105128 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.105129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:43.106479 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:43.106482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.107429 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.107430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:43.107501 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:43.107506 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:49:43.107537 (+ 31us) client_negotiation.cc:240] Negotiation successful
0807 22:49:43.107579 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":35,"spinlock_wait_cycles":768,"thread_start_us":101,"threads_started":1}
I20260807 22:49:43.108146 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:43.102735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50712 (local address 127.16.98.131:34955)
0807 22:49:43.102987 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:43.102992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:43.103336 (+ 344us) server_negotiation.cc:407] Connection header received
0807 22:49:43.103500 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:43.103502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:43.103521 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:43.103623 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:43.103793 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.104957 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.106588 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.107335 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.107874 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:43.107978 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:43.108007 (+ 29us) server_negotiation.cc:299] Negotiation successful
0807 22:49:43.108055 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":145,"threads_started":1}
I20260807 22:49:43.108702 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:43.102462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:44555 (local address 127.0.0.1:60052)
0807 22:49:43.103528 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:43.103539 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:43.103592 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:43.103852 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:43.103852 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:43.103854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:43.103888 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:43.103889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.104915 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.104916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:43.107137 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:43.107140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.108211 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.108212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:43.108485 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:43.108490 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 22:49:43.108591 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 22:49:43.108637 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":58,"thread_start_us":88,"threads_started":1}
I20260807 22:49:43.108915 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:43.102867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60052 (local address 127.16.98.129:44555)
0807 22:49:43.103075 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:43.103080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:43.103631 (+ 551us) server_negotiation.cc:407] Connection header received
0807 22:49:43.103683 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:43.103687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:43.103702 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:43.103780 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:43.103983 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.104824 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.107243 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.108028 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.108094 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:43.108171 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:43.108618 (+ 447us) server_negotiation.cc:299] Negotiation successful
0807 22:49:43.108666 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
I20260807 22:49:43.108799 4821 tablet_service.cc:1467] Processing CreateTablet for tablet 491475d6c31e4f6d81dfdb84abc40f5f (DEFAULT_TABLE table=test-table [id=3ec4ae3bef2d474facbce751eaec58c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:43.108877 4954 tablet_service.cc:1467] Processing CreateTablet for tablet 491475d6c31e4f6d81dfdb84abc40f5f (DEFAULT_TABLE table=test-table [id=3ec4ae3bef2d474facbce751eaec58c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:43.109207 4821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491475d6c31e4f6d81dfdb84abc40f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:43.109373 4954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491475d6c31e4f6d81dfdb84abc40f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:43.109618 4688 tablet_service.cc:1467] Processing CreateTablet for tablet 491475d6c31e4f6d81dfdb84abc40f5f (DEFAULT_TABLE table=test-table [id=3ec4ae3bef2d474facbce751eaec58c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 22:49:43.109966 4688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491475d6c31e4f6d81dfdb84abc40f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:43.112375 5034 tablet_bootstrap.cc:492] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb: Bootstrap starting.
I20260807 22:49:43.113197 5034 tablet_bootstrap.cc:654] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:43.113468 5035 tablet_bootstrap.cc:492] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f: Bootstrap starting.
I20260807 22:49:43.113569 5034 log.cc:824] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:43.113781 5036 tablet_bootstrap.cc:492] T 491475d6c31e4f6d81dfdb84abc40f5f P 64655259b3d74217b63f8257dd92c39f: Bootstrap starting.
I20260807 22:49:43.114446 5035 tablet_bootstrap.cc:654] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:43.114552 5036 tablet_bootstrap.cc:654] T 491475d6c31e4f6d81dfdb84abc40f5f P 64655259b3d74217b63f8257dd92c39f: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:43.114578 5034 tablet_bootstrap.cc:492] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb: No bootstrap required, opened a new log
I20260807 22:49:43.114645 5034 ts_tablet_manager.cc:1397] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 22:49:43.114804 5035 log.cc:824] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:43.114938 5036 log.cc:824] T 491475d6c31e4f6d81dfdb84abc40f5f P 64655259b3d74217b63f8257dd92c39f: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:43.115587 5035 tablet_bootstrap.cc:492] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f: No bootstrap required, opened a new log
I20260807 22:49:43.115671 5035 ts_tablet_manager.cc:1397] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260807 22:49:43.115844 5036 tablet_bootstrap.cc:492] T 491475d6c31e4f6d81dfdb84abc40f5f P 64655259b3d74217b63f8257dd92c39f: No bootstrap required, opened a new log
I20260807 22:49:43.115926 5036 ts_tablet_manager.cc:1397] T 491475d6c31e4f6d81dfdb84abc40f5f P 64655259b3d74217b63f8257dd92c39f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260807 22:49:43.117835 5034 raft_consensus.cc:348] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8e75e542d64598a5ba65cf151ad1fb" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34955 } } peers { permanent_uuid: "e7b91165e1d94f3ab55c120348f34d7f" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38181 } } peers { permanent_uuid: "64655259b3d74217b63f8257dd92c39f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44555 } }
I20260807 22:49:43.117900 5035 raft_consensus.cc:348] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8e75e542d64598a5ba65cf151ad1fb" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34955 } } peers { permanent_uuid: "e7b91165e1d94f3ab55c120348f34d7f" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38181 } } peers { permanent_uuid: "64655259b3d74217b63f8257dd92c39f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44555 } }
I20260807 22:49:43.118062 5034 raft_consensus.cc:374] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:43.118084 5035 raft_consensus.cc:374] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:43.118105 5034 raft_consensus.cc:729] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c8e75e542d64598a5ba65cf151ad1fb, State: Initialized, Role: FOLLOWER
I20260807 22:49:43.118137 5035 raft_consensus.cc:729] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7b91165e1d94f3ab55c120348f34d7f, State: Initialized, Role: FOLLOWER
I20260807 22:49:43.118292 5035 consensus_queue.cc:260] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8e75e542d64598a5ba65cf151ad1fb" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34955 } } peers { permanent_uuid: "e7b91165e1d94f3ab55c120348f34d7f" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38181 } } peers { permanent_uuid: "64655259b3d74217b63f8257dd92c39f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44555 } }
I20260807 22:49:43.118289 5034 consensus_queue.cc:260] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8e75e542d64598a5ba65cf151ad1fb" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34955 } } peers { permanent_uuid: "e7b91165e1d94f3ab55c120348f34d7f" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38181 } } peers { permanent_uuid: "64655259b3d74217b63f8257dd92c39f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44555 } }
I20260807 22:49:43.118606 5034 ts_tablet_manager.cc:1428] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260807 22:49:43.118604 5035 ts_tablet_manager.cc:1428] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 22:49:43.118752 5020 heartbeater.cc:503] Master 127.16.98.190:37323 was elected leader, sending a full tablet report...
I20260807 22:49:43.118774 4887 heartbeater.cc:503] Master 127.16.98.190:37323 was elected leader, sending a full tablet report...
I20260807 22:49:43.119026 5036 raft_consensus.cc:348] T 491475d6c31e4f6d81dfdb84abc40f5f P 64655259b3d74217b63f8257dd92c39f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8e75e542d64598a5ba65cf151ad1fb" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34955 } } peers { permanent_uuid: "e7b91165e1d94f3ab55c120348f34d7f" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38181 } } peers { permanent_uuid: "64655259b3d74217b63f8257dd92c39f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44555 } }
I20260807 22:49:43.119234 5036 raft_consensus.cc:374] T 491475d6c31e4f6d81dfdb84abc40f5f P 64655259b3d74217b63f8257dd92c39f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:43.119300 5036 raft_consensus.cc:729] T 491475d6c31e4f6d81dfdb84abc40f5f P 64655259b3d74217b63f8257dd92c39f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64655259b3d74217b63f8257dd92c39f, State: Initialized, Role: FOLLOWER
I20260807 22:49:43.119453 5036 consensus_queue.cc:260] T 491475d6c31e4f6d81dfdb84abc40f5f P 64655259b3d74217b63f8257dd92c39f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8e75e542d64598a5ba65cf151ad1fb" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34955 } } peers { permanent_uuid: "e7b91165e1d94f3ab55c120348f34d7f" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38181 } } peers { permanent_uuid: "64655259b3d74217b63f8257dd92c39f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44555 } }
I20260807 22:49:43.119747 5036 ts_tablet_manager.cc:1428] T 491475d6c31e4f6d81dfdb84abc40f5f P 64655259b3d74217b63f8257dd92c39f: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260807 22:49:43.119881 4754 heartbeater.cc:503] Master 127.16.98.190:37323 was elected leader, sending a full tablet report...
W20260807 22:49:43.197341 4888 tablet.cc:2369] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:49:43.256794 5041 raft_consensus.cc:482] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:49:43.257022 5041 raft_consensus.cc:504] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8e75e542d64598a5ba65cf151ad1fb" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34955 } } peers { permanent_uuid: "e7b91165e1d94f3ab55c120348f34d7f" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38181 } } peers { permanent_uuid: "64655259b3d74217b63f8257dd92c39f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44555 } }
I20260807 22:49:43.257449 5041 leader_election.cc:288] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c8e75e542d64598a5ba65cf151ad1fb (127.16.98.131:34955), 64655259b3d74217b63f8257dd92c39f (127.16.98.129:44555)
I20260807 22:49:43.261184 4889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:43.257688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.131:34955 (local address 127.16.98.130:41461)
0807 22:49:43.257775 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:43.257787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:43.257838 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:43.258070 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:43.258071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:43.258074 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:43.258164 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:43.258166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.259136 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.259138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:43.260091 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:43.260093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.260875 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.260876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:43.260962 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:43.260966 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 22:49:43.261028 (+ 62us) client_negotiation.cc:240] Negotiation successful
0807 22:49:43.261077 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 22:49:43.261169 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:43.257786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:41461 (local address 127.16.98.131:34955)
0807 22:49:43.257862 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:43.257864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:43.257897 (+ 33us) server_negotiation.cc:407] Connection header received
0807 22:49:43.257934 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:43.257936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:43.257948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:43.258006 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:43.258376 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.259025 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.260206 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.260767 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.260834 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:43.260842 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:43.261038 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 22:49:43.261091 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 22:49:43.261260 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:43.257839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.98.130:57773 (local address 127.16.98.129:44555)
0807 22:49:43.257892 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:43.257894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:43.258074 (+ 180us) server_negotiation.cc:407] Connection header received
0807 22:49:43.258110 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:43.258112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:43.258136 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:43.258223 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 22:49:43.258577 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.259228 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.260347 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.260840 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.260958 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:43.260967 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:43.261137 (+ 170us) server_negotiation.cc:299] Negotiation successful
0807 22:49:43.261182 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 22:49:43.261329 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:43.257733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:44555 (local address 127.16.98.130:57773)
0807 22:49:43.257925 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:43.257936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:49:43.257998 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:49:43.258374 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:49:43.258375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:49:43.258378 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:49:43.258453 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:43.258454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.259337 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.259338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:43.260193 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:49:43.260196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.260961 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.260962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:49:43.261054 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:43.261057 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 22:49:43.261121 (+ 64us) client_negotiation.cc:240] Negotiation successful
0807 22:49:43.261161 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":118,"threads_started":1}
I20260807 22:49:43.261767 4974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491475d6c31e4f6d81dfdb84abc40f5f" candidate_uuid: "e7b91165e1d94f3ab55c120348f34d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c8e75e542d64598a5ba65cf151ad1fb" is_pre_election: true
I20260807 22:49:43.261938 4974 raft_consensus.cc:2454] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7b91165e1d94f3ab55c120348f34d7f in term 0.
I20260807 22:49:43.262207 4774 leader_election.cc:302] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c8e75e542d64598a5ba65cf151ad1fb, e7b91165e1d94f3ab55c120348f34d7f; no voters:
I20260807 22:49:43.262272 4708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491475d6c31e4f6d81dfdb84abc40f5f" candidate_uuid: "e7b91165e1d94f3ab55c120348f34d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64655259b3d74217b63f8257dd92c39f" is_pre_election: true
I20260807 22:49:43.262404 4708 raft_consensus.cc:2454] T 491475d6c31e4f6d81dfdb84abc40f5f P 64655259b3d74217b63f8257dd92c39f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7b91165e1d94f3ab55c120348f34d7f in term 0.
I20260807 22:49:43.262388 5041 raft_consensus.cc:2781] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:49:43.262487 5041 raft_consensus.cc:482] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:49:43.262532 5041 raft_consensus.cc:3037] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:43.263321 5041 raft_consensus.cc:504] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8e75e542d64598a5ba65cf151ad1fb" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34955 } } peers { permanent_uuid: "e7b91165e1d94f3ab55c120348f34d7f" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38181 } } peers { permanent_uuid: "64655259b3d74217b63f8257dd92c39f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44555 } }
I20260807 22:49:43.263476 5041 leader_election.cc:288] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [CANDIDATE]: Term 1 election: Requested vote from peers 5c8e75e542d64598a5ba65cf151ad1fb (127.16.98.131:34955), 64655259b3d74217b63f8257dd92c39f (127.16.98.129:44555)
I20260807 22:49:43.263655 4708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491475d6c31e4f6d81dfdb84abc40f5f" candidate_uuid: "e7b91165e1d94f3ab55c120348f34d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64655259b3d74217b63f8257dd92c39f"
I20260807 22:49:43.263684 4974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491475d6c31e4f6d81dfdb84abc40f5f" candidate_uuid: "e7b91165e1d94f3ab55c120348f34d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c8e75e542d64598a5ba65cf151ad1fb"
I20260807 22:49:43.263736 4708 raft_consensus.cc:3037] T 491475d6c31e4f6d81dfdb84abc40f5f P 64655259b3d74217b63f8257dd92c39f [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:43.263784 4974 raft_consensus.cc:3037] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:43.264530 4974 raft_consensus.cc:2454] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7b91165e1d94f3ab55c120348f34d7f in term 1.
I20260807 22:49:43.264544 4708 raft_consensus.cc:2454] T 491475d6c31e4f6d81dfdb84abc40f5f P 64655259b3d74217b63f8257dd92c39f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7b91165e1d94f3ab55c120348f34d7f in term 1.
I20260807 22:49:43.264729 4774 leader_election.cc:302] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c8e75e542d64598a5ba65cf151ad1fb, e7b91165e1d94f3ab55c120348f34d7f; no voters:
I20260807 22:49:43.264837 5041 raft_consensus.cc:2781] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:43.265028 5041 raft_consensus.cc:686] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [term 1 LEADER]: Becoming Leader. State: Replica: e7b91165e1d94f3ab55c120348f34d7f, State: Running, Role: LEADER
I20260807 22:49:43.265136 5041 consensus_queue.cc:237] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8e75e542d64598a5ba65cf151ad1fb" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34955 } } peers { permanent_uuid: "e7b91165e1d94f3ab55c120348f34d7f" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38181 } } peers { permanent_uuid: "64655259b3d74217b63f8257dd92c39f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44555 } }
I20260807 22:49:43.266057 4561 catalog_manager.cc:5495] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f reported cstate change: term changed from 0 to 1, leader changed from <none> to e7b91165e1d94f3ab55c120348f34d7f (127.16.98.130). New cstate: current_term: 1 leader_uuid: "e7b91165e1d94f3ab55c120348f34d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8e75e542d64598a5ba65cf151ad1fb" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7b91165e1d94f3ab55c120348f34d7f" member_type: VOTER last_known_addr { host: "127.16.98.130" port: 38181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64655259b3d74217b63f8257dd92c39f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44555 } health_report { overall_health: UNKNOWN } } }
I20260807 22:49:43.290977 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:43.288059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.16.98.130:38181)
0807 22:49:43.288106 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:43.288108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:43.288116 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:49:43.288243 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:43.288244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:43.288265 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:43.288326 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:49:43.288596 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.289212 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.290067 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:43.290284 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:43.290376 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:43.290565 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:49:43.290680 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:49:43.290806 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:43.290828 (+ 22us) server_negotiation.cc:299] Negotiation successful
0807 22:49:43.290850 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260807 22:49:43.293049 4708 raft_consensus.cc:1261] T 491475d6c31e4f6d81dfdb84abc40f5f P 64655259b3d74217b63f8257dd92c39f [term 1 FOLLOWER]: Refusing update from remote peer e7b91165e1d94f3ab55c120348f34d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:49:43.293084 4974 raft_consensus.cc:1261] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb [term 1 FOLLOWER]: Refusing update from remote peer e7b91165e1d94f3ab55c120348f34d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:49:43.293500 5041 consensus_queue.cc:1035] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "64655259b3d74217b63f8257dd92c39f" member_type: VOTER last_known_addr { host: "127.16.98.129" port: 44555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:49:43.293715 5044 consensus_queue.cc:1035] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c8e75e542d64598a5ba65cf151ad1fb" member_type: VOTER last_known_addr { host: "127.16.98.131" port: 34955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:49:43.296825 5050 mvcc.cc:205] Tried to move back new op lower bound from 7316041659565154304 to 7316041659454885888. Current Snapshot: MvccSnapshot[applied={T|T < 7316041659565154304 or (T in {7316041659565154304})}]
I20260807 22:49:43.298544 4562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:49:43.298645 4562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 22:49:43.299722 4562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a3078c3d9e3499d85e7794f7441a62d: Sending DeleteTablet for 3 replicas of tablet 491475d6c31e4f6d81dfdb84abc40f5f
I20260807 22:49:43.300112 4954 tablet_service.cc:1514] Processing DeleteTablet for tablet 491475d6c31e4f6d81dfdb84abc40f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:43 UTC) from {username='slave'} at 127.0.0.1:50712
I20260807 22:49:43.300155 4688 tablet_service.cc:1514] Processing DeleteTablet for tablet 491475d6c31e4f6d81dfdb84abc40f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:43 UTC) from {username='slave'} at 127.0.0.1:60052
I20260807 22:49:43.300153 4821 tablet_service.cc:1514] Processing DeleteTablet for tablet 491475d6c31e4f6d81dfdb84abc40f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 22:49:43 UTC) from {username='slave'} at 127.0.0.1:60652
I20260807 22:49:43.300400 5058 tablet_replica.cc:330] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f: stopping tablet replica
I20260807 22:49:43.300509 5058 raft_consensus.cc:2229] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [term 1 LEADER]: Raft consensus shutting down.
I20260807 22:49:43.300662 5058 raft_consensus.cc:2258] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:49:43.301069 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 4625
I20260807 22:49:43.301358 5058 ts_tablet_manager.cc:1905] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 22:49:43.303771 5058 ts_tablet_manager.cc:1918] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:49:43.303858 5058 log.cc:1196] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-1/wal/wals/491475d6c31e4f6d81dfdb84abc40f5f
I20260807 22:49:43.304127 5058 ts_tablet_manager.cc:1939] T 491475d6c31e4f6d81dfdb84abc40f5f P e7b91165e1d94f3ab55c120348f34d7f: Deleting consensus metadata
I20260807 22:49:43.304863 4546 catalog_manager.cc:4841] TS e7b91165e1d94f3ab55c120348f34d7f (127.16.98.130:38181): tablet 491475d6c31e4f6d81dfdb84abc40f5f (table test-table [id=3ec4ae3bef2d474facbce751eaec58c6]) successfully deleted
I20260807 22:49:43.308733 5056 tablet_replica.cc:330] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb: stopping tablet replica
I20260807 22:49:43.308895 5056 raft_consensus.cc:2229] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 22:49:43.309026 5056 raft_consensus.cc:2258] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 22:49:43.309535 5056 ts_tablet_manager.cc:1905] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 22:49:43.310765 4549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.98.129:44555 (error 108)
I20260807 22:49:43.311515 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 4757
I20260807 22:49:43.311537 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:43.311367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.129:44555 (local address 127.0.0.1:60058)
0807 22:49:43.311439 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0807 22:49:43.311493 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.129:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 22:49:43.311568 5056 ts_tablet_manager.cc:1918] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 22:49:43.311645 5056 log.cc:1196] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb: Deleting WAL directory at /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786142716408518-16778-0/minicluster-data/ts-2/wal/wals/491475d6c31e4f6d81dfdb84abc40f5f
I20260807 22:49:43.312055 5056 ts_tablet_manager.cc:1939] T 491475d6c31e4f6d81dfdb84abc40f5f P 5c8e75e542d64598a5ba65cf151ad1fb: Deleting consensus metadata
W20260807 22:49:43.314004 4549 catalog_manager.cc:4584] TS 64655259b3d74217b63f8257dd92c39f (127.16.98.129:44555): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 491475d6c31e4f6d81dfdb84abc40f5f: Network error: Client connection negotiation failed: client connection to 127.16.98.129:44555: connect: Connection refused (error 111)
I20260807 22:49:43.314133 4547 catalog_manager.cc:4841] TS 5c8e75e542d64598a5ba65cf151ad1fb (127.16.98.131:34955): tablet 491475d6c31e4f6d81dfdb84abc40f5f (table test-table [id=3ec4ae3bef2d474facbce751eaec58c6]) successfully deleted
I20260807 22:49:43.319824 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 4890
I20260807 22:49:43.326203 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 4530
2026-08-07T22:49:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (882 ms)
[----------] 6 tests from EncryptionPolicyTest (9222 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:49:43 dist-test-slave-rgb1 krb5kdc[5063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:49:43 dist-test-slave-rgb1 krb5kdc[5063](info): set up 2 sockets
Aug 07 22:49:43 dist-test-slave-rgb1 krb5kdc[5063](info): commencing operation
krb5kdc: starting...
W20260807 22:49:44.462352 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:49:44 dist-test-slave-rgb1 krb5kdc[5063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:49:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 22:49:44.609434 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:44.696633 5076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:44.696846 5076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:44.696879 5076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:44.697988 5076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:44.698027 5076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:44.698045 5076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:44.698060 5076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:44.698071 5076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:44.699517 5076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46297
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:44.699719 5076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:44.699913 5076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:44.705056 5082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:44.705219 5085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:44.705080 5081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:44.705132 5076 server_base.cc:1034] running on GCE node
I20260807 22:49:44.705626 5076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:44.705951 5076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:44.707134 5076 hybrid_clock.cc:648] HybridClock initialized: now 1786142984707105 us; error 60 us; skew 500 ppm
Aug 07 22:49:44 dist-test-slave-rgb1 krb5kdc[5063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:44.715315 5076 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 22:49:44.715821 5076 webserver.cc:460] Webserver started at http://127.0.0.1:44849/ using document root <none> and password file <none>
I20260807 22:49:44.716094 5076 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:44.716171 5076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:44.716327 5076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:44.717514 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "c4818069c361435fb380ca2b8a331566"
format_stamp: "Formatted at 2026-08-07 22:49:44 on dist-test-slave-rgb1"
server_key: "6c5e69673728595323b81c005ff07de6"
server_key_iv: "468534e875cb04766a9d5dc3b8df194c"
server_key_version: "encryptionkey@0"
I20260807 22:49:44.717885 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "c4818069c361435fb380ca2b8a331566"
format_stamp: "Formatted at 2026-08-07 22:49:44 on dist-test-slave-rgb1"
server_key: "6c5e69673728595323b81c005ff07de6"
server_key_iv: "468534e875cb04766a9d5dc3b8df194c"
server_key_version: "encryptionkey@0"
I20260807 22:49:44.719331 5076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 22:49:44.720291 5092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:44.720670 5076 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:49:44.720781 5076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "c4818069c361435fb380ca2b8a331566"
format_stamp: "Formatted at 2026-08-07 22:49:44 on dist-test-slave-rgb1"
server_key: "6c5e69673728595323b81c005ff07de6"
server_key_iv: "468534e875cb04766a9d5dc3b8df194c"
server_key_version: "encryptionkey@0"
I20260807 22:49:44.720872 5076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:44.733569 5076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:44.736141 5076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:44.736356 5076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:44.741003 5076 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46297
I20260807 22:49:44.741062 5144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46297 every 8 connection(s)
I20260807 22:49:44.741492 5076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:44.742465 5146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:44.744937 5146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566: Bootstrap starting.
I20260807 22:49:44.745640 5146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:44.745839 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 5076
I20260807 22:49:44.745967 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:44.746011 5146 log.cc:824] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:44.746214 16778 external_mini_cluster.cc:1428] Setting key 4674434d1d0273790992362a75da57cc
I20260807 22:49:44.746909 5146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566: No bootstrap required, opened a new log
I20260807 22:49:44.748876 5146 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4818069c361435fb380ca2b8a331566" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46297 } }
I20260807 22:49:44.749032 5146 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:44.749071 5146 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4818069c361435fb380ca2b8a331566, State: Initialized, Role: FOLLOWER
I20260807 22:49:44.749274 5146 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4818069c361435fb380ca2b8a331566" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46297 } }
I20260807 22:49:44.749375 5146 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:44.749433 5146 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:44.749490 5146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:44.750494 5146 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4818069c361435fb380ca2b8a331566" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46297 } }
I20260807 22:49:44.750627 5146 leader_election.cc:302] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4818069c361435fb380ca2b8a331566; no voters:
I20260807 22:49:44.750875 5146 leader_election.cc:288] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:44.750908 5151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:44.751029 5151 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [term 1 LEADER]: Becoming Leader. State: Replica: c4818069c361435fb380ca2b8a331566, State: Running, Role: LEADER
I20260807 22:49:44.751175 5146 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:44.751217 5151 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4818069c361435fb380ca2b8a331566" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46297 } }
I20260807 22:49:44.751788 5153 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4818069c361435fb380ca2b8a331566. Latest consensus state: current_term: 1 leader_uuid: "c4818069c361435fb380ca2b8a331566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4818069c361435fb380ca2b8a331566" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46297 } } }
I20260807 22:49:44.751891 5153 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:44.752135 5152 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4818069c361435fb380ca2b8a331566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4818069c361435fb380ca2b8a331566" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46297 } } }
I20260807 22:49:44.752275 5152 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:44.752451 5160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:44.753110 5160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 07 22:49:44 dist-test-slave-rgb1 krb5kdc[5063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 22:49:44.755264 5160 catalog_manager.cc:1347] Generated new cluster ID: a48b4aebeea74dc6a7adac4f6e86b918
I20260807 22:49:44.755319 5160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:49:44.760176 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:44.746802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51543 (local address 127.0.0.1:46297)
0807 22:49:44.747127 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:44.747137 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:44.747175 (+ 38us) server_negotiation.cc:407] Connection header received
0807 22:49:44.748865 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:44.748874 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:44.748957 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:44.749137 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:44.749948 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:44.751011 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:44.751927 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:44.752605 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:44.752722 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:44.755778 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:44.755783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:44.755783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:44.755796 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:44.757899 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:44.758414 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:44.758423 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:44.758424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:44.758573 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:44.758837 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:44.758838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:44.758838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:44.759195 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:44.759368 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:44.759506 (+ 138us) server_negotiation.cc:299] Negotiation successful
0807 22:49:44.759744 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":158,"threads_started":1}
I20260807 22:49:44.761962 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:49:44.768780 5160 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:44.769325 5160 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:44.777622 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:44.763518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44957 (local address 127.0.0.1:46297)
0807 22:49:44.763554 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:44.763556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:44.768433 (+ 4877us) server_negotiation.cc:407] Connection header received
0807 22:49:44.768484 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:44.768486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:44.768510 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:44.768624 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:44.770086 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:44.770847 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:44.771912 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:44.772247 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:44.772312 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:44.773911 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:44.773913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:44.773913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:44.773921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:44.776268 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:44.776736 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:44.776737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:44.776737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:44.776799 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:44.777059 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:44.777060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:44.777060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:44.777188 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:44.777293 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:44.777447 (+ 154us) server_negotiation.cc:299] Negotiation successful
0807 22:49:44.777517 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260807 22:49:44.778090 5160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4818069c361435fb380ca2b8a331566: Generated new TSK 0
I20260807 22:49:44.778268 5160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:49:44.781528 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 5076
W20260807 22:49:44.787194 5060 connection.cc:535] client connection to 127.0.0.1:46297 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46297): Connection reset by peer (error 104)
2026-08-07T22:49:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:49:44 dist-test-slave-rgb1 krb5kdc[5177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:49:44 dist-test-slave-rgb1 krb5kdc[5177](info): set up 2 sockets
Aug 07 22:49:44 dist-test-slave-rgb1 krb5kdc[5177](info): commencing operation
krb5kdc: starting...
W20260807 22:49:45.924007 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:49:46 dist-test-slave-rgb1 krb5kdc[5177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:49:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 22:49:46.069053 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37957
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:46.159987 5190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:46.160208 5190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:46.160233 5190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:46.161451 5190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:46.161497 5190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:46.161514 5190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:46.161527 5190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:46.161540 5190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:46.163074 5190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37957
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:46.163350 5190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:46.163592 5190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:46.168891 5199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:46.168856 5197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:46.168895 5196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:46.169265 5190 server_base.cc:1034] running on GCE node
I20260807 22:49:46.169524 5190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:46.169835 5190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:46.171053 5190 hybrid_clock.cc:648] HybridClock initialized: now 1786142986171029 us; error 39 us; skew 500 ppm
Aug 07 22:49:46 dist-test-slave-rgb1 krb5kdc[5177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:46.174353 5190 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 22:49:46.174981 5190 webserver.cc:460] Webserver started at http://127.0.0.1:35737/ using document root <none> and password file <none>
I20260807 22:49:46.175248 5190 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:46.175326 5190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:46.175536 5190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:46.176841 5190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "b9103322fa3743eda3b072ecc5695389"
format_stamp: "Formatted at 2026-08-07 22:49:46 on dist-test-slave-rgb1"
server_key: "311c0acb5e95b0d684d77851ce907f35"
server_key_iv: "eeaba9d3a56ca62a8ebdf970c05e40ff"
server_key_version: "encryptionkey@0"
I20260807 22:49:46.177261 5190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "b9103322fa3743eda3b072ecc5695389"
format_stamp: "Formatted at 2026-08-07 22:49:46 on dist-test-slave-rgb1"
server_key: "311c0acb5e95b0d684d77851ce907f35"
server_key_iv: "eeaba9d3a56ca62a8ebdf970c05e40ff"
server_key_version: "encryptionkey@0"
I20260807 22:49:46.178850 5190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:49:46.179782 5206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:46.180013 5190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:46.180136 5190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "b9103322fa3743eda3b072ecc5695389"
format_stamp: "Formatted at 2026-08-07 22:49:46 on dist-test-slave-rgb1"
server_key: "311c0acb5e95b0d684d77851ce907f35"
server_key_iv: "eeaba9d3a56ca62a8ebdf970c05e40ff"
server_key_version: "encryptionkey@0"
I20260807 22:49:46.180224 5190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:46.193557 5190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:46.196153 5190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:46.196344 5190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:46.201056 5190 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37957
I20260807 22:49:46.201138 5261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37957 every 8 connection(s)
I20260807 22:49:46.201536 5190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:46.202836 5262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:46.205420 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 5190
I20260807 22:49:46.205540 5262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389: Bootstrap starting.
I20260807 22:49:46.205607 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:46.205878 16778 external_mini_cluster.cc:1428] Setting key 1b3620e174bf9afcaefd527be4ba551f
I20260807 22:49:46.206570 5262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:46.207221 5262 log.cc:824] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:46.208053 5262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389: No bootstrap required, opened a new log
I20260807 22:49:46.210350 5262 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9103322fa3743eda3b072ecc5695389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260807 22:49:46.210546 5262 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:46.210602 5262 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9103322fa3743eda3b072ecc5695389, State: Initialized, Role: FOLLOWER
I20260807 22:49:46.210773 5262 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9103322fa3743eda3b072ecc5695389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260807 22:49:46.210862 5262 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:46.210904 5262 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:46.210954 5262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:46.211649 5262 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9103322fa3743eda3b072ecc5695389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260807 22:49:46.211786 5262 leader_election.cc:302] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9103322fa3743eda3b072ecc5695389; no voters:
I20260807 22:49:46.212041 5262 leader_election.cc:288] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:46.212088 5267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:46.212114 5265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:46.206771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37953 (local address 127.0.0.1:37957)
0807 22:49:46.206947 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:46.206955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:46.206983 (+ 28us) server_negotiation.cc:407] Connection header received
0807 22:49:46.207541 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:46.207548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:46.207598 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:46.207755 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:46.208015 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:46.209067 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:46.209879 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:46.210289 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:46.210415 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:46.211401 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:46.211405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:46.211406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:46.211606 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:46.211649 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:46.211661 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:46.211763 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:46.211852 (+ 89us) server_negotiation.cc:299] Negotiation successful
0807 22:49:46.211880 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260807 22:49:46.212281 5267 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [term 1 LEADER]: Becoming Leader. State: Replica: b9103322fa3743eda3b072ecc5695389, State: Running, Role: LEADER
I20260807 22:49:46.212374 5262 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:46.212397 5267 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9103322fa3743eda3b072ecc5695389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260807 22:49:46.212983 5267 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9103322fa3743eda3b072ecc5695389. Latest consensus state: current_term: 1 leader_uuid: "b9103322fa3743eda3b072ecc5695389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9103322fa3743eda3b072ecc5695389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } } }
I20260807 22:49:46.212996 5269 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9103322fa3743eda3b072ecc5695389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9103322fa3743eda3b072ecc5695389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } } }
I20260807 22:49:46.213157 5267 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:46.213192 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:49:46.213171 5269 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389 [sys.catalog]: This master's current role is: LEADER
W20260807 22:49:46.214604 5283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:49:46.214674 5283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:49:46.214962 5276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:46.215642 5276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:46.217589 5276 catalog_manager.cc:1347] Generated new cluster ID: 44e2f89f86a848a3a0a4569ffcd126a0
I20260807 22:49:46.217650 5276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:49:46.219089 5265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:46.215119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49715 (local address 127.0.0.1:37957)
0807 22:49:46.215183 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:46.215185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:46.215314 (+ 129us) server_negotiation.cc:407] Connection header received
0807 22:49:46.215548 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:46.215550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:46.215564 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:46.215623 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:46.215827 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:46.216449 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:46.217554 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:46.217868 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:46.217949 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:46.218775 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:46.218776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:46.218777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:46.218782 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:46.218799 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:46.218803 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:46.218868 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:46.218974 (+ 106us) server_negotiation.cc:299] Negotiation successful
0807 22:49:46.218992 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260807 22:49:46.227957 5276 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:46.228632 5276 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:46.233428 5276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9103322fa3743eda3b072ecc5695389: Generated new TSK 0
I20260807 22:49:46.233659 5276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:49:46.240370 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 5190
W20260807 22:49:46.246280 5175 connection.cc:535] client connection to 127.0.0.1:37957 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37957): Connection reset by peer (error 104)
2026-08-07T22:49:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1458 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:49:46 dist-test-slave-rgb1 krb5kdc[5294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:49:46 dist-test-slave-rgb1 krb5kdc[5294](info): set up 2 sockets
Aug 07 22:49:46 dist-test-slave-rgb1 krb5kdc[5294](info): commencing operation
krb5kdc: starting...
W20260807 22:49:47.345100 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:49:47 dist-test-slave-rgb1 krb5kdc[5294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:49:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 22:49:47.505704 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:47.605383 5307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:47.605636 5307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:47.605666 5307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:47.606950 5307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:47.606995 5307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:47.607012 5307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:47.607026 5307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:47.607040 5307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:47.608716 5307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40585
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:47.608959 5307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:47.609176 5307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:47.615249 5312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:47.615249 5315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:47.615517 5307 server_base.cc:1034] running on GCE node
W20260807 22:49:47.615258 5313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:47.615823 5307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:47.616168 5307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:47.617347 5307 hybrid_clock.cc:648] HybridClock initialized: now 1786142987617319 us; error 46 us; skew 500 ppm
Aug 07 22:49:47 dist-test-slave-rgb1 krb5kdc[5294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:47.620317 5307 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 22:49:47.621084 5307 webserver.cc:460] Webserver started at http://127.0.0.1:43363/ using document root <none> and password file <none>
I20260807 22:49:47.621466 5307 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:47.621553 5307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:47.621721 5307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:47.623049 5307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "dcc6eefbac6443dda877623227dbe898"
format_stamp: "Formatted at 2026-08-07 22:49:47 on dist-test-slave-rgb1"
server_key: "8a6c773fb105e98c5fe6ac5cf1e9be96"
server_key_iv: "04cef2c3e7a2f3708e7a4d826bf48f88"
server_key_version: "encryptionkey@0"
I20260807 22:49:47.623446 5307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "dcc6eefbac6443dda877623227dbe898"
format_stamp: "Formatted at 2026-08-07 22:49:47 on dist-test-slave-rgb1"
server_key: "8a6c773fb105e98c5fe6ac5cf1e9be96"
server_key_iv: "04cef2c3e7a2f3708e7a4d826bf48f88"
server_key_version: "encryptionkey@0"
I20260807 22:49:47.625046 5307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260807 22:49:47.625959 5322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:47.626171 5307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:49:47.626269 5307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "dcc6eefbac6443dda877623227dbe898"
format_stamp: "Formatted at 2026-08-07 22:49:47 on dist-test-slave-rgb1"
server_key: "8a6c773fb105e98c5fe6ac5cf1e9be96"
server_key_iv: "04cef2c3e7a2f3708e7a4d826bf48f88"
server_key_version: "encryptionkey@0"
I20260807 22:49:47.626345 5307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:47.630170 5307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:47.632766 5307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:47.632889 5307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:47.637427 5307 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40585
I20260807 22:49:47.637459 5377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40585 every 8 connection(s)
I20260807 22:49:47.637858 5307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:47.638844 5378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:47.641335 5378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898: Bootstrap starting.
I20260807 22:49:47.641999 5378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:47.642302 5378 log.cc:824] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:47.642299 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 5307
I20260807 22:49:47.642511 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:47.642733 16778 external_mini_cluster.cc:1428] Setting key a0465d159b2fc3a675cc8676dbc394bc
I20260807 22:49:47.643157 5378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898: No bootstrap required, opened a new log
I20260807 22:49:47.645097 5378 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc6eefbac6443dda877623227dbe898" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40585 } }
I20260807 22:49:47.645268 5378 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:47.645334 5378 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc6eefbac6443dda877623227dbe898, State: Initialized, Role: FOLLOWER
I20260807 22:49:47.645494 5378 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc6eefbac6443dda877623227dbe898" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40585 } }
I20260807 22:49:47.645645 5378 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:47.645699 5378 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:47.645756 5378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:47.646744 5382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:47.643402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53525 (local address 127.0.0.1:40585)
0807 22:49:47.643640 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:47.643648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:47.643676 (+ 28us) server_negotiation.cc:407] Connection header received
0807 22:49:47.645086 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:47.645094 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:47.645132 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:47.645276 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:47.646177 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:47.646181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:47.646181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:47.646186 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:47.646230 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:47.646243 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:47.646315 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:47.646422 (+ 107us) server_negotiation.cc:299] Negotiation successful
0807 22:49:47.646494 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":132,"threads_started":1}
I20260807 22:49:47.646955 5378 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc6eefbac6443dda877623227dbe898" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40585 } }
I20260807 22:49:47.647066 5378 leader_election.cc:302] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcc6eefbac6443dda877623227dbe898; no voters:
I20260807 22:49:47.647269 5378 leader_election.cc:288] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:47.647294 5383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:47.647555 5383 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [term 1 LEADER]: Becoming Leader. State: Replica: dcc6eefbac6443dda877623227dbe898, State: Running, Role: LEADER
I20260807 22:49:47.647605 5378 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:47.647646 5383 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc6eefbac6443dda877623227dbe898" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40585 } }
I20260807 22:49:47.647823 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:49:47.648078 5383 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcc6eefbac6443dda877623227dbe898. Latest consensus state: current_term: 1 leader_uuid: "dcc6eefbac6443dda877623227dbe898" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc6eefbac6443dda877623227dbe898" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40585 } } }
I20260807 22:49:47.648170 5383 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:47.648336 5384 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcc6eefbac6443dda877623227dbe898" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc6eefbac6443dda877623227dbe898" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40585 } } }
I20260807 22:49:47.648411 5384 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:47.648524 5391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:47.649137 5391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:47.650923 5391 catalog_manager.cc:1347] Generated new cluster ID: e1950c8574de46269652d766d9c0ab1a
I20260807 22:49:47.650985 5391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:49:47.654492 5382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:47.653099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56169 (local address 127.0.0.1:40585)
0807 22:49:47.653155 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:47.653157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:47.653351 (+ 194us) server_negotiation.cc:407] Connection header received
0807 22:49:47.653420 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:47.653421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:47.653431 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:47.653495 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:47.654249 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:47.654250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:47.654251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:47.654255 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:47.654272 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:47.654277 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:47.654342 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:47.654405 (+ 63us) server_negotiation.cc:299] Negotiation successful
0807 22:49:47.654421 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 22:49:47.656260 5391 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:47.656450 5391 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:47.663955 5391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcc6eefbac6443dda877623227dbe898: Generated new TSK 0
I20260807 22:49:47.664186 5391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:49:47.669732 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 5307
2026-08-07T22:49:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:49:47 dist-test-slave-rgb1 krb5kdc[5410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:49:47 dist-test-slave-rgb1 krb5kdc[5410](info): set up 2 sockets
Aug 07 22:49:47 dist-test-slave-rgb1 krb5kdc[5410](info): commencing operation
krb5kdc: starting...
W20260807 22:49:48.826509 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:49:48 dist-test-slave-rgb1 krb5kdc[5410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:49:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 22:49:48.977227 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:49.067196 5423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:49.067456 5423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:49.067497 5423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:49.068727 5423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:49.068794 5423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:49.068820 5423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:49.068852 5423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:49.068887 5423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:49.070387 5423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:49.070660 5423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:49.070914 5423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:49.076272 5433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:49.076363 5423 server_base.cc:1034] running on GCE node
W20260807 22:49:49.076272 5428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:49.076298 5429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:49.076792 5423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:49.077109 5423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:49.078284 5423 hybrid_clock.cc:648] HybridClock initialized: now 1786142989078271 us; error 34 us; skew 500 ppm
Aug 07 22:49:49 dist-test-slave-rgb1 krb5kdc[5410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:49.081307 5423 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 22:49:49.081830 5423 webserver.cc:460] Webserver started at http://127.0.0.1:45775/ using document root <none> and password file <none>
I20260807 22:49:49.082046 5423 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:49.082093 5423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:49.082193 5423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:49.083323 5423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "c46cdeb29898411bb05d75811fabf712"
format_stamp: "Formatted at 2026-08-07 22:49:49 on dist-test-slave-rgb1"
server_key: "461e805c172153b5336ccb70683dfb24"
server_key_iv: "ad68593c2f6043b728697a54c0d517a5"
server_key_version: "encryptionkey@0"
I20260807 22:49:49.083662 5423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "c46cdeb29898411bb05d75811fabf712"
format_stamp: "Formatted at 2026-08-07 22:49:49 on dist-test-slave-rgb1"
server_key: "461e805c172153b5336ccb70683dfb24"
server_key_iv: "ad68593c2f6043b728697a54c0d517a5"
server_key_version: "encryptionkey@0"
I20260807 22:49:49.085253 5423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:49:49.086411 5440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:49.086659 5423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:49.086790 5423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "c46cdeb29898411bb05d75811fabf712"
format_stamp: "Formatted at 2026-08-07 22:49:49 on dist-test-slave-rgb1"
server_key: "461e805c172153b5336ccb70683dfb24"
server_key_iv: "ad68593c2f6043b728697a54c0d517a5"
server_key_version: "encryptionkey@0"
I20260807 22:49:49.086891 5423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:49.121599 5423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:49.124619 5423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:49.124928 5423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:49.129876 5423 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41043
I20260807 22:49:49.129957 5492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41043 every 8 connection(s)
I20260807 22:49:49.130366 5423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:49.131436 5493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:49.134248 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 5423
I20260807 22:49:49.134433 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:49.134673 16778 external_mini_cluster.cc:1428] Setting key 6c34aa763d0b799f1946e15a4217d10e
I20260807 22:49:49.135257 5493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712: Bootstrap starting.
I20260807 22:49:49.136006 5493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:49.136410 5493 log.cc:824] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:49.137413 5493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712: No bootstrap required, opened a new log
I20260807 22:49:49.139942 5493 raft_consensus.cc:348] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46cdeb29898411bb05d75811fabf712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41043 } }
I20260807 22:49:49.140112 5493 raft_consensus.cc:374] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:49.140141 5493 raft_consensus.cc:729] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c46cdeb29898411bb05d75811fabf712, State: Initialized, Role: FOLLOWER
I20260807 22:49:49.140316 5493 consensus_queue.cc:260] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46cdeb29898411bb05d75811fabf712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41043 } }
I20260807 22:49:49.140436 5493 raft_consensus.cc:388] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:49.140533 5493 raft_consensus.cc:482] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:49.140638 5493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:49.141369 5493 raft_consensus.cc:504] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46cdeb29898411bb05d75811fabf712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41043 } }
I20260807 22:49:49.141476 5493 leader_election.cc:302] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c46cdeb29898411bb05d75811fabf712; no voters:
I20260807 22:49:49.141642 5493 leader_election.cc:288] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:49.141739 5498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:49.141935 5498 raft_consensus.cc:686] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [term 1 LEADER]: Becoming Leader. State: Replica: c46cdeb29898411bb05d75811fabf712, State: Running, Role: LEADER
I20260807 22:49:49.142053 5498 consensus_queue.cc:237] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46cdeb29898411bb05d75811fabf712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41043 } }
I20260807 22:49:49.142096 5493 sys_catalog.cc:564] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:49.142580 5500 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c46cdeb29898411bb05d75811fabf712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46cdeb29898411bb05d75811fabf712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41043 } } }
I20260807 22:49:49.142709 5500 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:49.143484 5507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:49.143721 5501 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c46cdeb29898411bb05d75811fabf712. Latest consensus state: current_term: 1 leader_uuid: "c46cdeb29898411bb05d75811fabf712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46cdeb29898411bb05d75811fabf712" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41043 } } }
I20260807 22:49:49.143818 5501 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:49.144320 5507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 07 22:49:49 dist-test-slave-rgb1 krb5kdc[5410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 22:49:49.147032 5507 catalog_manager.cc:1347] Generated new cluster ID: 1b1e2fb85abf472cb717ace1fe86133c
I20260807 22:49:49.147106 5507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:49:49.150993 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:49.135316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42287 (local address 127.0.0.1:41043)
0807 22:49:49.135546 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:49.135554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:49.135582 (+ 28us) server_negotiation.cc:407] Connection header received
0807 22:49:49.136105 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:49.136116 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:49.136184 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:49.136322 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:49.137299 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:49.138403 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:49.139725 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:49.140293 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:49.140418 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:49.145555 (+ 5137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:49.145559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:49.145560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:49.145575 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:49.148107 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:49.148709 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:49.148713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:49.148713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:49.148841 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:49.149129 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:49.149130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:49.149130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:49.149497 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:49.149673 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:49.150388 (+ 715us) server_negotiation.cc:299] Negotiation successful
0807 22:49:49.150561 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":105,"threads_started":1}
I20260807 22:49:49.152640 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:49:49.164948 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:49.154281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45805 (local address 127.0.0.1:41043)
0807 22:49:49.154330 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:49.154332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:49.156768 (+ 2436us) server_negotiation.cc:407] Connection header received
0807 22:49:49.156843 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:49.156844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:49.156856 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:49.156923 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:49.157665 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:49.158481 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:49.159599 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:49.159942 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:49.160011 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:49.161702 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:49.161704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:49.161704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:49.161713 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:49.163456 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:49.163895 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:49.163897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:49.163897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:49.163961 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:49.164187 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:49.164188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:49.164188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:49.164328 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:49.164429 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:49.164592 (+ 163us) server_negotiation.cc:299] Negotiation successful
0807 22:49:49.164834 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260807 22:49:49.180526 5507 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:49.181288 5507 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:49.186699 5507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c46cdeb29898411bb05d75811fabf712: Generated new TSK 0
I20260807 22:49:49.186973 5507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:49:49.193161 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 5423
2026-08-07T22:49:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1522 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:49:49 dist-test-slave-rgb1 krb5kdc[5527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:49:49 dist-test-slave-rgb1 krb5kdc[5527](info): set up 2 sockets
Aug 07 22:49:49 dist-test-slave-rgb1 krb5kdc[5527](info): commencing operation
krb5kdc: starting...
W20260807 22:49:50.312726 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:49:50 dist-test-slave-rgb1 krb5kdc[5527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 22:49:50.461551 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36361
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:50.556639 5540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:50.556921 5540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:50.556952 5540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:50.559597 5540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:50.559788 5540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:50.560053 5540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:50.560113 5540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:50.560150 5540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:50.562212 5540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:50.562525 5540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:50.562841 5540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:50.568531 5545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:50.568691 5546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:50.568882 5540 server_base.cc:1034] running on GCE node
W20260807 22:49:50.568531 5548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:50.569352 5540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:50.569717 5540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:50.570928 5540 hybrid_clock.cc:648] HybridClock initialized: now 1786142990570945 us; error 90 us; skew 500 ppm
Aug 07 22:49:50 dist-test-slave-rgb1 krb5kdc[5527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:50.578485 5540 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 22:49:50.579053 5540 webserver.cc:460] Webserver started at http://127.0.0.1:41871/ using document root <none> and password file <none>
I20260807 22:49:50.579332 5540 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:50.579406 5540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:50.579566 5540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:50.580801 5540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "f0f74c621dd64474bb72cb9693e74de3"
format_stamp: "Formatted at 2026-08-07 22:49:50 on dist-test-slave-rgb1"
server_key: "4cd84dd8926af9f64f737c23d11f7ae6"
server_key_iv: "37eddbb57d622806e339066de76a119d"
server_key_version: "encryptionkey@0"
I20260807 22:49:50.581210 5540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "f0f74c621dd64474bb72cb9693e74de3"
format_stamp: "Formatted at 2026-08-07 22:49:50 on dist-test-slave-rgb1"
server_key: "4cd84dd8926af9f64f737c23d11f7ae6"
server_key_iv: "37eddbb57d622806e339066de76a119d"
server_key_version: "encryptionkey@0"
I20260807 22:49:50.582793 5540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:49:50.584004 5555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:50.584206 5540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:50.584319 5540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "f0f74c621dd64474bb72cb9693e74de3"
format_stamp: "Formatted at 2026-08-07 22:49:50 on dist-test-slave-rgb1"
server_key: "4cd84dd8926af9f64f737c23d11f7ae6"
server_key_iv: "37eddbb57d622806e339066de76a119d"
server_key_version: "encryptionkey@0"
I20260807 22:49:50.584410 5540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:50.615176 5540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:50.617568 5540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:50.617733 5540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:50.622556 5540 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36361
I20260807 22:49:50.622735 5610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36361 every 8 connection(s)
I20260807 22:49:50.623081 5540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:50.624449 5611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:50.628142 5611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3: Bootstrap starting.
I20260807 22:49:50.628804 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 5540
I20260807 22:49:50.628912 5611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:50.628947 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:50.629171 16778 external_mini_cluster.cc:1428] Setting key 66f267f2b840d3dc65595609fb3550cc
I20260807 22:49:50.629302 5611 log.cc:824] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:50.630201 5611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3: No bootstrap required, opened a new log
I20260807 22:49:50.632663 5611 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f74c621dd64474bb72cb9693e74de3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36361 } }
I20260807 22:49:50.632835 5611 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:50.632866 5611 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f74c621dd64474bb72cb9693e74de3, State: Initialized, Role: FOLLOWER
I20260807 22:49:50.632992 5611 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f74c621dd64474bb72cb9693e74de3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36361 } }
I20260807 22:49:50.633076 5611 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:50.633118 5611 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:50.633183 5611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:50.633908 5611 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f74c621dd64474bb72cb9693e74de3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36361 } }
I20260807 22:49:50.634042 5611 leader_election.cc:302] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0f74c621dd64474bb72cb9693e74de3; no voters:
I20260807 22:49:50.634240 5611 leader_election.cc:288] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:50.634320 5617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:50.634485 5617 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [term 1 LEADER]: Becoming Leader. State: Replica: f0f74c621dd64474bb72cb9693e74de3, State: Running, Role: LEADER
I20260807 22:49:50.634552 5611 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:50.634606 5617 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f74c621dd64474bb72cb9693e74de3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36361 } }
I20260807 22:49:50.634992 5618 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0f74c621dd64474bb72cb9693e74de3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f74c621dd64474bb72cb9693e74de3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36361 } } }
I20260807 22:49:50.635308 5618 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:50.635217 5619 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0f74c621dd64474bb72cb9693e74de3. Latest consensus state: current_term: 1 leader_uuid: "f0f74c621dd64474bb72cb9693e74de3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f74c621dd64474bb72cb9693e74de3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36361 } } }
I20260807 22:49:50.636042 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:50.629775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36517 (local address 127.0.0.1:36361)
0807 22:49:50.629987 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:50.629997 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:50.630036 (+ 39us) server_negotiation.cc:407] Connection header received
0807 22:49:50.630564 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:50.630572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:50.630623 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:50.630748 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:50.631005 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:50.631923 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:50.633133 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:50.633608 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:50.633700 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:50.635216 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:50.635220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:50.635220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:50.635442 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:50.635484 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:50.635500 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:50.635555 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:50.635707 (+ 152us) server_negotiation.cc:299] Negotiation successful
0807 22:49:50.635797 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":125,"threads_started":1}
I20260807 22:49:50.635805 5619 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:50.636278 5626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:50.636803 5626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:50.636981 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:49:50.638597 5626 catalog_manager.cc:1347] Generated new cluster ID: 07e9f51d535e4eaaa9b7102b3506c2d8
I20260807 22:49:50.638657 5626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:49:50.642028 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:50.638458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55435 (local address 127.0.0.1:36361)
0807 22:49:50.638516 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:50.638518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:50.638567 (+ 49us) server_negotiation.cc:407] Connection header received
0807 22:49:50.638656 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:50.638657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:50.638667 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:50.638727 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:50.638915 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:50.639627 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:50.640681 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:50.640971 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:50.641034 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:50.641787 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:50.641788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:50.641788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:50.641793 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:50.641808 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:50.641812 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:50.641861 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:50.641933 (+ 72us) server_negotiation.cc:299] Negotiation successful
0807 22:49:50.641953 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260807 22:49:50.645689 5626 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:50.646322 5626 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:50.659380 5626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0f74c621dd64474bb72cb9693e74de3: Generated new TSK 0
I20260807 22:49:50.659622 5626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:49:50.668293 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 5540
2026-08-07T22:49:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1475 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:49:50 dist-test-slave-rgb1 krb5kdc[5643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:49:50 dist-test-slave-rgb1 krb5kdc[5643](info): set up 2 sockets
Aug 07 22:49:50 dist-test-slave-rgb1 krb5kdc[5643](info): commencing operation
krb5kdc: starting...
W20260807 22:49:51.804306 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:49:51 dist-test-slave-rgb1 krb5kdc[5643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:49:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 22:49:51.945442 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:52.032675 5656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:52.032887 5656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:52.032912 5656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:52.034096 5656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:52.034145 5656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:52.034159 5656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:52.034173 5656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:52.034184 5656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:52.035743 5656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:52.036021 5656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:52.036269 5656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:52.041409 5664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:52.041399 5662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:52.041466 5661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:52.041766 5656 server_base.cc:1034] running on GCE node
I20260807 22:49:52.042028 5656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:52.042315 5656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:52.043484 5656 hybrid_clock.cc:648] HybridClock initialized: now 1786142992043470 us; error 34 us; skew 500 ppm
Aug 07 22:49:52 dist-test-slave-rgb1 krb5kdc[5643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:52.046429 5656 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 22:49:52.046868 5656 webserver.cc:460] Webserver started at http://127.0.0.1:43979/ using document root <none> and password file <none>
I20260807 22:49:52.047075 5656 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:52.047119 5656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:52.047217 5656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:52.048336 5656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "50225752e5cf4506bf36efc5801bb39d"
format_stamp: "Formatted at 2026-08-07 22:49:52 on dist-test-slave-rgb1"
server_key: "22fa64df64e9231e0c52427959516223"
server_key_iv: "f92bb21be12a8211775a1c21d2c08148"
server_key_version: "encryptionkey@0"
I20260807 22:49:52.048763 5656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "50225752e5cf4506bf36efc5801bb39d"
format_stamp: "Formatted at 2026-08-07 22:49:52 on dist-test-slave-rgb1"
server_key: "22fa64df64e9231e0c52427959516223"
server_key_iv: "f92bb21be12a8211775a1c21d2c08148"
server_key_version: "encryptionkey@0"
I20260807 22:49:52.050225 5656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:49:52.051182 5671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:52.051456 5656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:49:52.051575 5656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "50225752e5cf4506bf36efc5801bb39d"
format_stamp: "Formatted at 2026-08-07 22:49:52 on dist-test-slave-rgb1"
server_key: "22fa64df64e9231e0c52427959516223"
server_key_iv: "f92bb21be12a8211775a1c21d2c08148"
server_key_version: "encryptionkey@0"
I20260807 22:49:52.051640 5656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:52.055354 5656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:52.057859 5656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:52.057984 5656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:52.062573 5656 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44549
I20260807 22:49:52.062606 5726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44549 every 8 connection(s)
I20260807 22:49:52.062988 5656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:52.064265 5727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:52.066763 5727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d: Bootstrap starting.
I20260807 22:49:52.067427 5727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:52.067808 5727 log.cc:824] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:52.069211 5727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d: No bootstrap required, opened a new log
I20260807 22:49:52.070890 5727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50225752e5cf4506bf36efc5801bb39d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44549 } }
I20260807 22:49:52.071066 5727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:52.071115 5727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50225752e5cf4506bf36efc5801bb39d, State: Initialized, Role: FOLLOWER
I20260807 22:49:52.071276 5727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50225752e5cf4506bf36efc5801bb39d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44549 } }
I20260807 22:49:52.071363 5727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:52.071408 5727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:52.071458 5727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:52.071925 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 5656
I20260807 22:49:52.072034 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:52.072045 5727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50225752e5cf4506bf36efc5801bb39d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44549 } }
I20260807 22:49:52.072166 5727 leader_election.cc:302] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50225752e5cf4506bf36efc5801bb39d; no voters:
I20260807 22:49:52.072300 16778 external_mini_cluster.cc:1428] Setting key 08d04ef54ec3093426786853737b4809
I20260807 22:49:52.072381 5727 leader_election.cc:288] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:52.072422 5730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:52.072589 5730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [term 1 LEADER]: Becoming Leader. State: Replica: 50225752e5cf4506bf36efc5801bb39d, State: Running, Role: LEADER
I20260807 22:49:52.072683 5727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:52.072692 5730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50225752e5cf4506bf36efc5801bb39d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44549 } }
I20260807 22:49:52.073058 5730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50225752e5cf4506bf36efc5801bb39d. Latest consensus state: current_term: 1 leader_uuid: "50225752e5cf4506bf36efc5801bb39d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50225752e5cf4506bf36efc5801bb39d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44549 } } }
I20260807 22:49:52.073146 5730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:52.073091 5732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50225752e5cf4506bf36efc5801bb39d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50225752e5cf4506bf36efc5801bb39d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44549 } } }
I20260807 22:49:52.073329 5732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:52.073374 5737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:52.073956 5737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:52.075454 5749 mvcc.cc:205] Tried to move back new op lower bound from 7316041695539261440 to 7316041695530254336. Current Snapshot: MvccSnapshot[applied={T|T < 7316041695539261440}]
I20260807 22:49:52.075518 5741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:52.073326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59969 (local address 127.0.0.1:44549)
0807 22:49:52.073767 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:52.073777 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:52.073810 (+ 33us) server_negotiation.cc:407] Connection header received
0807 22:49:52.073927 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:52.073934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:52.073964 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:52.074429 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:52.074935 (+ 506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:52.074939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:52.074939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:52.074943 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:52.074984 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:52.074998 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:52.075051 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:52.075271 (+ 220us) server_negotiation.cc:299] Negotiation successful
0807 22:49:52.075312 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":88,"threads_started":1}
I20260807 22:49:52.075956 5737 catalog_manager.cc:1347] Generated new cluster ID: ec684b39704448e690fe812660bcb245
I20260807 22:49:52.076002 5737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:49:52.076442 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:49:52.081861 5741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:52.078050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40523 (local address 127.0.0.1:44549)
0807 22:49:52.078110 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:52.078112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:52.080762 (+ 2650us) server_negotiation.cc:407] Connection header received
0807 22:49:52.080844 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:52.080845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:52.080851 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:52.080907 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:52.081611 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:52.081612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:52.081613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:52.081617 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:52.081633 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:52.081638 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:52.081688 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:52.081751 (+ 63us) server_negotiation.cc:299] Negotiation successful
0807 22:49:52.081768 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 22:49:52.089478 5737 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:52.089777 5737 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:52.092634 5737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50225752e5cf4506bf36efc5801bb39d: Generated new TSK 0
I20260807 22:49:52.092876 5737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:49:52.101049 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 5656
2026-08-07T22:49:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:49:52 dist-test-slave-rgb1 krb5kdc[5758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:49:52 dist-test-slave-rgb1 krb5kdc[5758](info): set up 2 sockets
Aug 07 22:49:52 dist-test-slave-rgb1 krb5kdc[5758](info): commencing operation
krb5kdc: starting...
W20260807 22:49:53.240921 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:49:53 dist-test-slave-rgb1 krb5kdc[5758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 22:49:53.380167 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:53.471019 5771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:53.471277 5771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:53.471306 5771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:53.472718 5771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:53.472769 5771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:53.472786 5771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:53.472800 5771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:53.472815 5771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:53.474489 5771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:53.474720 5771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:53.474920 5771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:53.480824 5777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:53.480860 5779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:53.480851 5776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:53.481160 5771 server_base.cc:1034] running on GCE node
I20260807 22:49:53.481448 5771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:53.481735 5771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:53.482949 5771 hybrid_clock.cc:648] HybridClock initialized: now 1786142993482912 us; error 59 us; skew 500 ppm
Aug 07 22:49:53 dist-test-slave-rgb1 krb5kdc[5758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:53.486187 5771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 22:49:53.486759 5771 webserver.cc:460] Webserver started at http://127.0.0.1:41709/ using document root <none> and password file <none>
I20260807 22:49:53.487072 5771 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:53.487161 5771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:53.487330 5771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:53.488631 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "0e5ac2ea20d544448fadd37df48bc24b"
format_stamp: "Formatted at 2026-08-07 22:49:53 on dist-test-slave-rgb1"
server_key: "61e46282f089c9b119272e4dba5ee05f"
server_key_iv: "6cdb868cb7713825447711713ef54fae"
server_key_version: "encryptionkey@0"
I20260807 22:49:53.489069 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "0e5ac2ea20d544448fadd37df48bc24b"
format_stamp: "Formatted at 2026-08-07 22:49:53 on dist-test-slave-rgb1"
server_key: "61e46282f089c9b119272e4dba5ee05f"
server_key_iv: "6cdb868cb7713825447711713ef54fae"
server_key_version: "encryptionkey@0"
I20260807 22:49:53.490576 5771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:49:53.491524 5786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:53.491849 5771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:53.491979 5771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "0e5ac2ea20d544448fadd37df48bc24b"
format_stamp: "Formatted at 2026-08-07 22:49:53 on dist-test-slave-rgb1"
server_key: "61e46282f089c9b119272e4dba5ee05f"
server_key_iv: "6cdb868cb7713825447711713ef54fae"
server_key_version: "encryptionkey@0"
I20260807 22:49:53.492139 5771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:53.522182 5771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:53.524672 5771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:53.524871 5771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:53.529556 5771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45127
I20260807 22:49:53.529649 5840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45127 every 8 connection(s)
I20260807 22:49:53.530072 5771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:53.531065 5841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:53.533474 5841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b: Bootstrap starting.
I20260807 22:49:53.534104 5841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:53.534415 5841 log.cc:824] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:53.535231 5841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b: No bootstrap required, opened a new log
I20260807 22:49:53.536906 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 5771
I20260807 22:49:53.537053 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:53.537284 16778 external_mini_cluster.cc:1428] Setting key 4bce48a8daa3e39b330d04679074ca75
I20260807 22:49:53.537215 5841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ac2ea20d544448fadd37df48bc24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260807 22:49:53.537388 5841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:53.537460 5841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5ac2ea20d544448fadd37df48bc24b, State: Initialized, Role: FOLLOWER
I20260807 22:49:53.537624 5841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ac2ea20d544448fadd37df48bc24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260807 22:49:53.537704 5841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:53.537747 5841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:53.537797 5841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:53.538862 5841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ac2ea20d544448fadd37df48bc24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260807 22:49:53.538988 5841 leader_election.cc:302] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e5ac2ea20d544448fadd37df48bc24b; no voters:
I20260807 22:49:53.539191 5841 leader_election.cc:288] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:53.539316 5846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:53.539510 5841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:53.539536 5846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [term 1 LEADER]: Becoming Leader. State: Replica: 0e5ac2ea20d544448fadd37df48bc24b, State: Running, Role: LEADER
I20260807 22:49:53.539739 5846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ac2ea20d544448fadd37df48bc24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260807 22:49:53.540251 5848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e5ac2ea20d544448fadd37df48bc24b. Latest consensus state: current_term: 1 leader_uuid: "0e5ac2ea20d544448fadd37df48bc24b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ac2ea20d544448fadd37df48bc24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } } }
I20260807 22:49:53.540216 5847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e5ac2ea20d544448fadd37df48bc24b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ac2ea20d544448fadd37df48bc24b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } } }
I20260807 22:49:53.540314 5848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:53.540325 5847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:53.540740 5857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:53.541513 5857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:53.543385 5857 catalog_manager.cc:1347] Generated new cluster ID: e63d1e00ba9548cbb6129318ec7693c1
I20260807 22:49:53.543442 5857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 07 22:49:53 dist-test-slave-rgb1 krb5kdc[5758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 22:49:53.549602 5857 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:53.550307 5857 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:53.551177 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:53.538145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:57291 (local address 127.0.0.1:45127)
0807 22:49:53.538401 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:53.538409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:53.538448 (+ 39us) server_negotiation.cc:407] Connection header received
0807 22:49:53.538557 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:53.538563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:53.538609 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:53.538745 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:53.539662 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:53.541220 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:53.542219 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:53.542836 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:53.542960 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:53.546867 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:53.546871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:53.546872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:53.546886 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:53.549143 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:53.549705 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:53.549709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:53.549709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:53.549817 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:53.550103 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:53.550104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:53.550105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:53.550378 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:53.550548 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:53.550825 (+ 277us) server_negotiation.cc:299] Negotiation successful
0807 22:49:53.550937 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":125,"threads_started":1}
I20260807 22:49:53.552243 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:49:53.561311 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:53.553857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:33781 (local address 127.0.0.1:45127)
0807 22:49:53.553896 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:53.553898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:53.553948 (+ 50us) server_negotiation.cc:407] Connection header received
0807 22:49:53.554015 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:53.554026 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:53.554057 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:53.554116 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:53.554742 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:53.555557 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:53.556580 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:53.556910 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:53.556977 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:53.558405 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:53.558407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:53.558407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:53.558415 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:53.560037 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:53.560461 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:53.560463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:53.560463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:53.560524 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:53.560785 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:53.560786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:53.560786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:53.560922 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:53.561038 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:53.561146 (+ 108us) server_negotiation.cc:299] Negotiation successful
0807 22:49:53.561214 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260807 22:49:53.562654 5857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e5ac2ea20d544448fadd37df48bc24b: Generated new TSK 0
I20260807 22:49:53.562836 5857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:49:53.570149 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 5771
W20260807 22:49:53.577464 5756 connection.cc:535] client connection to 127.0.0.1:45127 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45127): Connection reset by peer (error 104)
2026-08-07T22:49:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:49:53 dist-test-slave-rgb1 krb5kdc[5875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:49:53 dist-test-slave-rgb1 krb5kdc[5875](info): set up 2 sockets
Aug 07 22:49:53 dist-test-slave-rgb1 krb5kdc[5875](info): commencing operation
krb5kdc: starting...
W20260807 22:49:54.684749 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:49:54 dist-test-slave-rgb1 krb5kdc[5875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:49:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 22:49:54.826844 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:54.918128 5889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:54.918383 5889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:54.918426 5889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:54.919937 5889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:54.920042 5889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:54.920070 5889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:54.920094 5889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:54.920125 5889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:54.921953 5889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:54.922302 5889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:54.922569 5889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:54.927877 5897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:54.927966 5895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:54.927923 5894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:54.928694 5889 server_base.cc:1034] running on GCE node
I20260807 22:49:54.928969 5889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:54.929315 5889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:54.930503 5889 hybrid_clock.cc:648] HybridClock initialized: now 1786142994930498 us; error 61 us; skew 500 ppm
Aug 07 22:49:54 dist-test-slave-rgb1 krb5kdc[5875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:54.934584 5889 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 22:49:54.935426 5889 webserver.cc:460] Webserver started at http://127.0.0.1:42039/ using document root <none> and password file <none>
I20260807 22:49:54.935702 5889 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:54.935778 5889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:54.935941 5889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:54.937424 5889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "3ddfe31759da49f1a7dfaa2c1ac5a451"
format_stamp: "Formatted at 2026-08-07 22:49:54 on dist-test-slave-rgb1"
server_key: "7679f7167f6ace723e4405f175409500"
server_key_iv: "d38d2ccafa74b957123d3da204cd56ed"
server_key_version: "encryptionkey@0"
I20260807 22:49:54.937894 5889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "3ddfe31759da49f1a7dfaa2c1ac5a451"
format_stamp: "Formatted at 2026-08-07 22:49:54 on dist-test-slave-rgb1"
server_key: "7679f7167f6ace723e4405f175409500"
server_key_iv: "d38d2ccafa74b957123d3da204cd56ed"
server_key_version: "encryptionkey@0"
I20260807 22:49:54.939474 5889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 22:49:54.940789 5904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:54.941190 5889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:54.941431 5889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "3ddfe31759da49f1a7dfaa2c1ac5a451"
format_stamp: "Formatted at 2026-08-07 22:49:54 on dist-test-slave-rgb1"
server_key: "7679f7167f6ace723e4405f175409500"
server_key_iv: "d38d2ccafa74b957123d3da204cd56ed"
server_key_version: "encryptionkey@0"
I20260807 22:49:54.941562 5889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:54.957523 5889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:54.960165 5889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:54.960427 5889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:54.965806 5889 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34253
I20260807 22:49:54.965840 5957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34253 every 8 connection(s)
I20260807 22:49:54.966251 5889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:54.967257 5958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:54.969799 5958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451: Bootstrap starting.
I20260807 22:49:54.970374 5958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:54.970638 5958 log.cc:824] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:54.971345 5958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451: No bootstrap required, opened a new log
I20260807 22:49:54.973059 5958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ddfe31759da49f1a7dfaa2c1ac5a451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260807 22:49:54.973201 5958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:54.973223 5958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ddfe31759da49f1a7dfaa2c1ac5a451, State: Initialized, Role: FOLLOWER
I20260807 22:49:54.973362 5958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ddfe31759da49f1a7dfaa2c1ac5a451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260807 22:49:54.973448 5958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:54.973471 5958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:54.973500 5958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:54.974009 5958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ddfe31759da49f1a7dfaa2c1ac5a451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260807 22:49:54.974098 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 5889
I20260807 22:49:54.974095 5958 leader_election.cc:302] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ddfe31759da49f1a7dfaa2c1ac5a451; no voters:
I20260807 22:49:54.974247 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:54.974341 5958 leader_election.cc:288] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:54.974530 16778 external_mini_cluster.cc:1428] Setting key 5c53dd3c5540e458146e2fdb5f6abf2a
I20260807 22:49:54.974534 5962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:54.974637 5958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:54.974659 5962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [term 1 LEADER]: Becoming Leader. State: Replica: 3ddfe31759da49f1a7dfaa2c1ac5a451, State: Running, Role: LEADER
I20260807 22:49:54.974814 5962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ddfe31759da49f1a7dfaa2c1ac5a451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } }
I20260807 22:49:54.975376 5963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ddfe31759da49f1a7dfaa2c1ac5a451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ddfe31759da49f1a7dfaa2c1ac5a451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } } }
I20260807 22:49:54.975487 5963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:54.975505 5964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ddfe31759da49f1a7dfaa2c1ac5a451. Latest consensus state: current_term: 1 leader_uuid: "3ddfe31759da49f1a7dfaa2c1ac5a451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ddfe31759da49f1a7dfaa2c1ac5a451" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34253 } } }
I20260807 22:49:54.975559 5964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:54.975750 5973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:54.976366 5973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:54.978039 5973 catalog_manager.cc:1347] Generated new cluster ID: 84e399a717ac483c860bb677f31ad849
I20260807 22:49:54.978097 5973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:49:54.980784 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:54.975158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:37543 (local address 127.0.0.1:34253)
0807 22:49:54.975672 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:54.975682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:54.975717 (+ 35us) server_negotiation.cc:407] Connection header received
0807 22:49:54.975839 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:54.975846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:54.975906 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:54.976357 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:54.976453 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:54.977880 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:54.978777 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:54.979205 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:54.979324 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:54.980076 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:54.980080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:54.980080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:54.980272 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:54.980303 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:54.980314 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:54.980410 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:54.980511 (+ 101us) server_negotiation.cc:299] Negotiation successful
0807 22:49:54.980542 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":79,"threads_started":1}
I20260807 22:49:54.981747 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:49:54.988332 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:54.983178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:49387 (local address 127.0.0.1:34253)
0807 22:49:54.983231 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:54.983233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:54.984784 (+ 1551us) server_negotiation.cc:407] Connection header received
0807 22:49:54.984866 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:54.984868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:54.984880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:54.984929 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:54.985061 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:54.985608 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:54.986841 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:54.987110 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:54.987195 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:54.988025 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:54.988027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:54.988027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:54.988032 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:54.988048 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:54.988053 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:54.988117 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:54.988198 (+ 81us) server_negotiation.cc:299] Negotiation successful
0807 22:49:54.988213 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260807 22:49:54.993598 5973 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:54.994479 5973 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:55.003443 5973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ddfe31759da49f1a7dfaa2c1ac5a451: Generated new TSK 0
I20260807 22:49:55.003674 5973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:49:55.008991 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 5889
W20260807 22:49:55.015273 5873 connection.cc:535] client connection to 127.0.0.1:34253 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34253): Connection reset by peer (error 104)
2026-08-07T22:49:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:49:55 dist-test-slave-rgb1 krb5kdc[5990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:49:55 dist-test-slave-rgb1 krb5kdc[5990](info): set up 2 sockets
Aug 07 22:49:55 dist-test-slave-rgb1 krb5kdc[5990](info): commencing operation
krb5kdc: starting...
W20260807 22:49:56.106600 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:49:56 dist-test-slave-rgb1 krb5kdc[5990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:49:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 22:49:56.249563 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:56.342041 6003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:56.342321 6003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:56.342366 6003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:56.343663 6003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:56.343741 6003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:56.343767 6003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:56.343793 6003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:56.343827 6003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:56.345575 6003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:56.345901 6003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:56.346172 6003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:56.351824 6008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:56.352234 6011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:56.352243 6009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:56.352931 6003 server_base.cc:1034] running on GCE node
I20260807 22:49:56.353257 6003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:56.353638 6003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:56.354960 6003 hybrid_clock.cc:648] HybridClock initialized: now 1786142996354927 us; error 43 us; skew 500 ppm
Aug 07 22:49:56 dist-test-slave-rgb1 krb5kdc[5990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:56.363770 6003 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 22:49:56.364372 6003 webserver.cc:460] Webserver started at http://127.0.0.1:39671/ using document root <none> and password file <none>
I20260807 22:49:56.364691 6003 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:56.364748 6003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:56.364904 6003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:56.366266 6003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "94a55ca895634d5ab2828df37dec1757"
format_stamp: "Formatted at 2026-08-07 22:49:56 on dist-test-slave-rgb1"
server_key: "16c25cb4d95b0eb6b8ce26288caa6d8a"
server_key_iv: "ef94348e01361e9448e9cbe7cfa56922"
server_key_version: "encryptionkey@0"
I20260807 22:49:56.366776 6003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "94a55ca895634d5ab2828df37dec1757"
format_stamp: "Formatted at 2026-08-07 22:49:56 on dist-test-slave-rgb1"
server_key: "16c25cb4d95b0eb6b8ce26288caa6d8a"
server_key_iv: "ef94348e01361e9448e9cbe7cfa56922"
server_key_version: "encryptionkey@0"
I20260807 22:49:56.368387 6003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:49:56.369508 6018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:56.369822 6003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:56.370033 6003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "94a55ca895634d5ab2828df37dec1757"
format_stamp: "Formatted at 2026-08-07 22:49:56 on dist-test-slave-rgb1"
server_key: "16c25cb4d95b0eb6b8ce26288caa6d8a"
server_key_iv: "ef94348e01361e9448e9cbe7cfa56922"
server_key_version: "encryptionkey@0"
I20260807 22:49:56.370131 6003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:56.374084 6003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:56.376837 6003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:56.377027 6003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:56.382166 6003 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39269
I20260807 22:49:56.382230 6074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39269 every 8 connection(s)
I20260807 22:49:56.382609 6003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:56.383553 6075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:56.386018 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 6003
I20260807 22:49:56.386157 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:56.386175 6075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757: Bootstrap starting.
I20260807 22:49:56.386359 16778 external_mini_cluster.cc:1428] Setting key 3ce8769ef371249c92e40c02a68047a0
I20260807 22:49:56.386952 6075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:56.387342 6075 log.cc:824] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:56.388324 6075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757: No bootstrap required, opened a new log
I20260807 22:49:56.389273 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:56.386898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:38207 (local address 127.0.0.1:39269)
0807 22:49:56.387106 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:56.387114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:56.387140 (+ 26us) server_negotiation.cc:407] Connection header received
0807 22:49:56.387721 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:56.387730 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:56.387775 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:56.387925 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:56.388801 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:56.388805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:56.388806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:56.388810 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:56.388861 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:56.388871 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:56.388987 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:56.389074 (+ 87us) server_negotiation.cc:299] Negotiation successful
0807 22:49:56.389125 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260807 22:49:56.390424 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:49:56.391183 6075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a55ca895634d5ab2828df37dec1757" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39269 } }
I20260807 22:49:56.391402 6075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:56.391453 6075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a55ca895634d5ab2828df37dec1757, State: Initialized, Role: FOLLOWER
I20260807 22:49:56.391587 6075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a55ca895634d5ab2828df37dec1757" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39269 } }
I20260807 22:49:56.391721 6075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:56.391826 6075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:56.391906 6075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:56.392928 6075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a55ca895634d5ab2828df37dec1757" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39269 } }
I20260807 22:49:56.393065 6075 leader_election.cc:302] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94a55ca895634d5ab2828df37dec1757; no voters:
I20260807 22:49:56.393268 6075 leader_election.cc:288] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:56.393385 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:56.392231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:48941 (local address 127.0.0.1:39269)
0807 22:49:56.392271 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:56.392273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:56.392360 (+ 87us) server_negotiation.cc:407] Connection header received
0807 22:49:56.392425 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:56.392426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:56.392430 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:56.392475 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:56.393064 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:56.393066 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:56.393066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:56.393070 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:56.393089 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:56.393095 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:56.393156 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:56.393264 (+ 108us) server_negotiation.cc:299] Negotiation successful
0807 22:49:56.393278 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260807 22:49:56.393395 6085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:56.393569 6085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [term 1 LEADER]: Becoming Leader. State: Replica: 94a55ca895634d5ab2828df37dec1757, State: Running, Role: LEADER
I20260807 22:49:56.393577 6075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:56.393764 6085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a55ca895634d5ab2828df37dec1757" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39269 } }
I20260807 22:49:56.394245 6087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94a55ca895634d5ab2828df37dec1757. Latest consensus state: current_term: 1 leader_uuid: "94a55ca895634d5ab2828df37dec1757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a55ca895634d5ab2828df37dec1757" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39269 } } }
I20260807 22:49:56.394348 6087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:56.394577 6086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94a55ca895634d5ab2828df37dec1757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a55ca895634d5ab2828df37dec1757" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39269 } } }
I20260807 22:49:56.394682 6086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:56.394992 6095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:56.395596 6095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:56.397205 6095 catalog_manager.cc:1347] Generated new cluster ID: 60a81f90a0f545de99d7a879ee44f84d
I20260807 22:49:56.397264 6095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:49:56.413336 6095 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:56.413546 6095 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:56.422978 6095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94a55ca895634d5ab2828df37dec1757: Generated new TSK 0
I20260807 22:49:56.423245 6095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:49:56.434547 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 6003
2026-08-07T22:49:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:49:56 dist-test-slave-rgb1 krb5kdc[6107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:49:56 dist-test-slave-rgb1 krb5kdc[6107](info): set up 2 sockets
Aug 07 22:49:56 dist-test-slave-rgb1 krb5kdc[6107](info): commencing operation
krb5kdc: starting...
W20260807 22:49:57.594779 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:49:57 dist-test-slave-rgb1 krb5kdc[6107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 22:49:57.735721 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:57.821265 6120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:57.821547 6120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:57.821593 6120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:57.822793 6120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:57.822865 6120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:57.822891 6120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:57.822919 6120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:57.822952 6120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:57.824505 6120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44435
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:57.824828 6120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:57.825083 6120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:57.830619 6126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:57.830601 6128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:57.830601 6125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:57.830972 6120 server_base.cc:1034] running on GCE node
I20260807 22:49:57.831192 6120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:57.831545 6120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:57.832856 6120 hybrid_clock.cc:648] HybridClock initialized: now 1786142997832777 us; error 87 us; skew 500 ppm
Aug 07 22:49:57 dist-test-slave-rgb1 krb5kdc[6107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:57.836264 6120 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 22:49:57.836840 6120 webserver.cc:460] Webserver started at http://127.0.0.1:41967/ using document root <none> and password file <none>
I20260807 22:49:57.837077 6120 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:57.837122 6120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:57.837220 6120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:57.838274 6120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "9c37f3863f514afca981a24a4f2a3597"
format_stamp: "Formatted at 2026-08-07 22:49:57 on dist-test-slave-rgb1"
server_key: "d513dbddc9fd217baf752c24a0b56258"
server_key_iv: "3e1d402c3ee5224c8daa88cb4c339ca1"
server_key_version: "encryptionkey@0"
I20260807 22:49:57.838583 6120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "9c37f3863f514afca981a24a4f2a3597"
format_stamp: "Formatted at 2026-08-07 22:49:57 on dist-test-slave-rgb1"
server_key: "d513dbddc9fd217baf752c24a0b56258"
server_key_iv: "3e1d402c3ee5224c8daa88cb4c339ca1"
server_key_version: "encryptionkey@0"
I20260807 22:49:57.840021 6120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:49:57.841053 6136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:57.841234 6120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:49:57.841315 6120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "9c37f3863f514afca981a24a4f2a3597"
format_stamp: "Formatted at 2026-08-07 22:49:57 on dist-test-slave-rgb1"
server_key: "d513dbddc9fd217baf752c24a0b56258"
server_key_iv: "3e1d402c3ee5224c8daa88cb4c339ca1"
server_key_version: "encryptionkey@0"
I20260807 22:49:57.841383 6120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:57.861133 6120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:57.863718 6120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:57.863860 6120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:57.869253 6120 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44435
I20260807 22:49:57.869629 6190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44435 every 8 connection(s)
I20260807 22:49:57.869797 6120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:57.870724 6191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:57.872237 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 6120
I20260807 22:49:57.872332 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:57.872649 16778 external_mini_cluster.cc:1428] Setting key ff39f1f7e3d70b51855f060e8a9f4872
I20260807 22:49:57.873548 6191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597: Bootstrap starting.
I20260807 22:49:57.874223 6191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:57.874591 6191 log.cc:824] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:57.875433 6191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597: No bootstrap required, opened a new log
I20260807 22:49:57.877915 6191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c37f3863f514afca981a24a4f2a3597" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44435 } }
I20260807 22:49:57.878108 6191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:57.878152 6191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c37f3863f514afca981a24a4f2a3597, State: Initialized, Role: FOLLOWER
I20260807 22:49:57.878299 6191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c37f3863f514afca981a24a4f2a3597" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44435 } }
I20260807 22:49:57.878388 6191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:57.878420 6191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:57.878471 6191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:57.879314 6191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c37f3863f514afca981a24a4f2a3597" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44435 } }
I20260807 22:49:57.879426 6191 leader_election.cc:302] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c37f3863f514afca981a24a4f2a3597; no voters:
I20260807 22:49:57.879635 6191 leader_election.cc:288] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:57.879731 6196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:57.879875 6196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [term 1 LEADER]: Becoming Leader. State: Replica: 9c37f3863f514afca981a24a4f2a3597, State: Running, Role: LEADER
I20260807 22:49:57.879932 6191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:57.879958 6196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c37f3863f514afca981a24a4f2a3597" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44435 } }
I20260807 22:49:57.880323 6196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c37f3863f514afca981a24a4f2a3597. Latest consensus state: current_term: 1 leader_uuid: "9c37f3863f514afca981a24a4f2a3597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c37f3863f514afca981a24a4f2a3597" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44435 } } }
Aug 07 22:49:57 dist-test-slave-rgb1 krb5kdc[6107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 22:49:57.880399 6196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:57.880332 6197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c37f3863f514afca981a24a4f2a3597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c37f3863f514afca981a24a4f2a3597" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44435 } } }
I20260807 22:49:57.880442 6197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:57.880795 6203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:57.881364 6203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:57.883700 6203 catalog_manager.cc:1347] Generated new cluster ID: 7f6a2d83be8d436390a9cf5d7b2de530
I20260807 22:49:57.883750 6203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:49:57.885306 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:57.873221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:56957 (local address 127.0.0.1:44435)
0807 22:49:57.873403 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:57.873411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:57.873438 (+ 27us) server_negotiation.cc:407] Connection header received
0807 22:49:57.873900 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:57.873907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:57.873957 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:57.874077 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:57.875010 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:57.876161 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:57.877282 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:57.877744 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:57.877838 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:57.881104 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:57.881108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:57.881109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:57.881122 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:57.883494 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:57.883989 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:57.883992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:57.883993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:57.884094 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:57.884353 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:57.884354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:57.884354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:57.884684 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:57.884831 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:57.884996 (+ 165us) server_negotiation.cc:299] Negotiation successful
0807 22:49:57.885093 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260807 22:49:57.893878 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:49:57.909482 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:57.895367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:45605 (local address 127.0.0.1:44435)
0807 22:49:57.895406 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:57.895408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:57.901176 (+ 5768us) server_negotiation.cc:407] Connection header received
0807 22:49:57.901314 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:57.901316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:57.901329 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:57.901415 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:57.902178 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:57.903335 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:57.904356 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:57.904707 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:57.904783 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:57.906410 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:57.906411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:57.906412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:49:57.906420 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:57.908137 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:57.908599 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:57.908600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:57.908600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:57.908664 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:49:57.908902 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:49:57.908903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:49:57.908903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:49:57.909068 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:57.909183 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:57.909311 (+ 128us) server_negotiation.cc:299] Negotiation successful
0807 22:49:57.909380 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260807 22:49:57.919956 6203 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:57.920608 6203 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:57.925747 6203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c37f3863f514afca981a24a4f2a3597: Generated new TSK 0
I20260807 22:49:57.926071 6203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:49:57.928826 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 6120
W20260807 22:49:57.936304 6105 connection.cc:535] client connection to 127.0.0.1:44435 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44435): Connection reset by peer (error 104)
2026-08-07T22:49:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1495 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:49:57 dist-test-slave-rgb1 krb5kdc[6224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:49:57 dist-test-slave-rgb1 krb5kdc[6224](info): set up 2 sockets
Aug 07 22:49:57 dist-test-slave-rgb1 krb5kdc[6224](info): commencing operation
krb5kdc: starting...
W20260807 22:49:59.021742 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:49:59 dist-test-slave-rgb1 krb5kdc[6224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 22:49:59.165326 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:49:59.251272 6238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:49:59.251531 6238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:49:59.251570 6238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:49:59.252797 6238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:49:59.252871 6238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:49:59.252895 6238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:49:59.252925 6238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:49:59.252956 6238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:49:59.254540 6238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40285
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:49:59.254838 6238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:49:59.255097 6238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:49:59.260443 6247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:59.260488 6243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:49:59.260434 6244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:49:59.260736 6238 server_base.cc:1034] running on GCE node
I20260807 22:49:59.261052 6238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:49:59.261396 6238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:49:59.262569 6238 hybrid_clock.cc:648] HybridClock initialized: now 1786142999262539 us; error 51 us; skew 500 ppm
Aug 07 22:49:59 dist-test-slave-rgb1 krb5kdc[6224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786142999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:49:59.265801 6238 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 22:49:59.266332 6238 webserver.cc:460] Webserver started at http://127.0.0.1:37603/ using document root <none> and password file <none>
I20260807 22:49:59.266608 6238 fs_manager.cc:359] Metadata directory not provided
I20260807 22:49:59.266661 6238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:49:59.266808 6238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:49:59.268004 6238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "71c81dd582f948709f63687ce971fb79"
format_stamp: "Formatted at 2026-08-07 22:49:59 on dist-test-slave-rgb1"
server_key: "6b152ba293b0847a441ec97aa343f473"
server_key_iv: "4360e869753c0166e8995920a5978df1"
server_key_version: "encryptionkey@0"
I20260807 22:49:59.268407 6238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "71c81dd582f948709f63687ce971fb79"
format_stamp: "Formatted at 2026-08-07 22:49:59 on dist-test-slave-rgb1"
server_key: "6b152ba293b0847a441ec97aa343f473"
server_key_iv: "4360e869753c0166e8995920a5978df1"
server_key_version: "encryptionkey@0"
I20260807 22:49:59.270093 6238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:49:59.271102 6254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:49:59.271446 6238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:49:59.271566 6238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "71c81dd582f948709f63687ce971fb79"
format_stamp: "Formatted at 2026-08-07 22:49:59 on dist-test-slave-rgb1"
server_key: "6b152ba293b0847a441ec97aa343f473"
server_key_iv: "4360e869753c0166e8995920a5978df1"
server_key_version: "encryptionkey@0"
I20260807 22:49:59.271677 6238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:49:59.310177 6238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:49:59.312726 6238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:49:59.312912 6238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:49:59.317377 6238 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40285
I20260807 22:49:59.317446 6306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40285 every 8 connection(s)
I20260807 22:49:59.317835 6238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:49:59.318758 6307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:49:59.321178 6307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79: Bootstrap starting.
I20260807 22:49:59.321738 6307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79: Neither blocks nor log segments found. Creating new log.
I20260807 22:49:59.322008 6307 log.cc:824] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79: Log is configured to *not* fsync() on all Append() calls
I20260807 22:49:59.322505 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 6238
I20260807 22:49:59.322664 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:49:59.322875 16778 external_mini_cluster.cc:1428] Setting key 413f0188b99aae506e34e3508969de59
I20260807 22:49:59.322870 6307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79: No bootstrap required, opened a new log
I20260807 22:49:59.326035 6307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c81dd582f948709f63687ce971fb79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } }
I20260807 22:49:59.326216 6307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:49:59.326246 6307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c81dd582f948709f63687ce971fb79, State: Initialized, Role: FOLLOWER
I20260807 22:49:59.326366 6307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c81dd582f948709f63687ce971fb79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } }
I20260807 22:49:59.326460 6307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:49:59.326498 6307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:49:59.326555 6307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:49:59.327178 6307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c81dd582f948709f63687ce971fb79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } }
I20260807 22:49:59.327303 6307 leader_election.cc:302] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71c81dd582f948709f63687ce971fb79; no voters:
I20260807 22:49:59.327459 6307 leader_election.cc:288] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:49:59.327522 6313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:49:59.327713 6307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:49:59.327732 6313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [term 1 LEADER]: Becoming Leader. State: Replica: 71c81dd582f948709f63687ce971fb79, State: Running, Role: LEADER
I20260807 22:49:59.327934 6313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c81dd582f948709f63687ce971fb79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } }
I20260807 22:49:59.328380 6314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71c81dd582f948709f63687ce971fb79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c81dd582f948709f63687ce971fb79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } } }
I20260807 22:49:59.328459 6314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:59.328935 6315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71c81dd582f948709f63687ce971fb79. Latest consensus state: current_term: 1 leader_uuid: "71c81dd582f948709f63687ce971fb79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c81dd582f948709f63687ce971fb79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } } }
I20260807 22:49:59.329099 6315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79 [sys.catalog]: This master's current role is: LEADER
I20260807 22:49:59.329293 6322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:49:59.329713 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:59.323445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:55191 (local address 127.0.0.1:40285)
0807 22:49:59.323673 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:59.323683 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:59.323720 (+ 37us) server_negotiation.cc:407] Connection header received
0807 22:49:59.324434 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:59.324443 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:59.324510 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:59.324703 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:59.324924 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:59.326358 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:59.327276 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:59.327896 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:59.328053 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:59.328984 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:59.328988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:59.328988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:59.329218 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:59.329261 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:59.329276 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:59.329365 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:59.329512 (+ 147us) server_negotiation.cc:299] Negotiation successful
0807 22:49:59.329541 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":100,"threads_started":1}
I20260807 22:49:59.329967 6322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:49:59.330770 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:49:59.332993 6322 catalog_manager.cc:1347] Generated new cluster ID: ad98bca688944532904e08b691af7b83
I20260807 22:49:59.333112 6322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:49:59.336043 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:49:59.332398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:45009 (local address 127.0.0.1:40285)
0807 22:49:59.332440 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0807 22:49:59.332442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:49:59.332593 (+ 151us) server_negotiation.cc:407] Connection header received
0807 22:49:59.332644 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:49:59.332645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:49:59.332656 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:49:59.332725 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:49:59.332871 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:59.333574 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:59.334637 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:49:59.334940 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:49:59.334997 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:49:59.335794 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:49:59.335795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:49:59.335796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:49:59.335800 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:49:59.335812 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:49:59.335816 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:49:59.335861 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0807 22:49:59.335963 (+ 102us) server_negotiation.cc:299] Negotiation successful
0807 22:49:59.335972 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260807 22:49:59.358960 6322 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:49:59.359606 6322 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:49:59.363060 6322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71c81dd582f948709f63687ce971fb79: Generated new TSK 0
I20260807 22:49:59.363281 6322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:49:59.368847 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 6238
W20260807 22:49:59.374995 6222 connection.cc:535] client connection to 127.0.0.1:40285 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40285): Connection reset by peer (error 104)
2026-08-07T22:49:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:49:59 dist-test-slave-rgb1 krb5kdc[6339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:49:59 dist-test-slave-rgb1 krb5kdc[6339](info): set up 2 sockets
Aug 07 22:49:59 dist-test-slave-rgb1 krb5kdc[6339](info): commencing operation
krb5kdc: starting...
W20260807 22:50:00.506705 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:50:00 dist-test-slave-rgb1 krb5kdc[6339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:50:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:50:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 22:50:00.675554 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:50:00.777022 6353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:50:00.777299 6353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:50:00.777331 6353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:50:00.779366 6353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:50:00.779491 6353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:50:00.779508 6353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:50:00.779523 6353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:50:00.779536 6353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:50:00.782351 6353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:50:00.782657 6353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:50:00.782879 6353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:50:00.789093 6358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:00.789251 6353 server_base.cc:1034] running on GCE node
W20260807 22:50:00.789167 6360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:00.789094 6362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:00.789634 6353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:50:00.789942 6353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:50:00.791205 6353 hybrid_clock.cc:648] HybridClock initialized: now 1786143000791164 us; error 67 us; skew 500 ppm
Aug 07 22:50:00 dist-test-slave-rgb1 krb5kdc[6339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:50:00.794334 6353 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 22:50:00.794790 6353 webserver.cc:460] Webserver started at http://127.0.0.1:36041/ using document root <none> and password file <none>
I20260807 22:50:00.795056 6353 fs_manager.cc:359] Metadata directory not provided
I20260807 22:50:00.795132 6353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:50:00.795295 6353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:50:00.796445 6353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "757b8bfcbeea43c3a7af84de78871b95"
format_stamp: "Formatted at 2026-08-07 22:50:00 on dist-test-slave-rgb1"
server_key: "8ce925977050d5dc28b05f78030bc0d8"
server_key_iv: "966dfeaecd4aa97eac13f4118fe2abaf"
server_key_version: "encryptionkey@0"
I20260807 22:50:00.796929 6353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "757b8bfcbeea43c3a7af84de78871b95"
format_stamp: "Formatted at 2026-08-07 22:50:00 on dist-test-slave-rgb1"
server_key: "8ce925977050d5dc28b05f78030bc0d8"
server_key_iv: "966dfeaecd4aa97eac13f4118fe2abaf"
server_key_version: "encryptionkey@0"
I20260807 22:50:00.798542 6353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:50:00.799649 6369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:50:00.800035 6353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:50:00.800170 6353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "757b8bfcbeea43c3a7af84de78871b95"
format_stamp: "Formatted at 2026-08-07 22:50:00 on dist-test-slave-rgb1"
server_key: "8ce925977050d5dc28b05f78030bc0d8"
server_key_iv: "966dfeaecd4aa97eac13f4118fe2abaf"
server_key_version: "encryptionkey@0"
I20260807 22:50:00.800266 6353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:50:00.804328 6353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:50:00.807065 6353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:50:00.807266 6353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:50:00.812840 6353 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35101
I20260807 22:50:00.812889 6423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35101 every 8 connection(s)
I20260807 22:50:00.813284 6353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:50:00.814294 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:50:00.815274 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 6353
I20260807 22:50:00.815431 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:50:00.815729 16778 external_mini_cluster.cc:1428] Setting key a6c30fbd5a7afff6029a75522921eaf2
I20260807 22:50:00.817584 6424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95: Bootstrap starting.
I20260807 22:50:00.818388 6424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95: Neither blocks nor log segments found. Creating new log.
I20260807 22:50:00.818758 6424 log.cc:824] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95: Log is configured to *not* fsync() on all Append() calls
I20260807 22:50:00.818955 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:00.816402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:48823 (local address 127.0.0.1:35101)
0807 22:50:00.816719 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:00.816729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:00.816758 (+ 29us) server_negotiation.cc:407] Connection header received
0807 22:50:00.817387 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:00.817395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:00.817426 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:00.817569 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:00.818338 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:00.818341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:00.818341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:50:00.818345 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:00.818389 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:50:00.818400 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:00.818531 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:00.818608 (+ 77us) server_negotiation.cc:299] Negotiation successful
0807 22:50:00.818686 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":85,"threads_started":1}
I20260807 22:50:00.819561 6424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95: No bootstrap required, opened a new log
I20260807 22:50:00.820158 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:50:00.822295 6424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757b8bfcbeea43c3a7af84de78871b95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35101 } }
I20260807 22:50:00.822499 6424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:50:00.822544 6424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757b8bfcbeea43c3a7af84de78871b95, State: Initialized, Role: FOLLOWER
I20260807 22:50:00.822844 6424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757b8bfcbeea43c3a7af84de78871b95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35101 } }
I20260807 22:50:00.823146 6424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:50:00.823220 6424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:50:00.823294 6424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:50:00.823618 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:00.821926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:41277 (local address 127.0.0.1:35101)
0807 22:50:00.821976 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:00.821978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:00.822064 (+ 86us) server_negotiation.cc:407] Connection header received
0807 22:50:00.822122 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:00.822123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:00.822129 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:00.822191 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:00.823279 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:00.823281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:00.823281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:50:00.823286 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:00.823303 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:50:00.823308 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:00.823360 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:00.823527 (+ 167us) server_negotiation.cc:299] Negotiation successful
0807 22:50:00.823540 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 22:50:00.824851 6424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757b8bfcbeea43c3a7af84de78871b95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35101 } }
I20260807 22:50:00.825047 6424 leader_election.cc:302] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 757b8bfcbeea43c3a7af84de78871b95; no voters:
I20260807 22:50:00.825344 6424 leader_election.cc:288] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:50:00.825519 6436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:50:00.825832 6436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [term 1 LEADER]: Becoming Leader. State: Replica: 757b8bfcbeea43c3a7af84de78871b95, State: Running, Role: LEADER
I20260807 22:50:00.826004 6424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:50:00.825996 6436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757b8bfcbeea43c3a7af84de78871b95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35101 } }
I20260807 22:50:00.826462 6437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "757b8bfcbeea43c3a7af84de78871b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757b8bfcbeea43c3a7af84de78871b95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35101 } } }
I20260807 22:50:00.826485 6438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 757b8bfcbeea43c3a7af84de78871b95. Latest consensus state: current_term: 1 leader_uuid: "757b8bfcbeea43c3a7af84de78871b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "757b8bfcbeea43c3a7af84de78871b95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35101 } } }
I20260807 22:50:00.826583 6437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:00.826592 6438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95 [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:00.826805 6442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:50:00.827339 6442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:50:00.829068 6442 catalog_manager.cc:1347] Generated new cluster ID: 13b31a0106784d35bcec1cc27b12a91d
I20260807 22:50:00.829133 6442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:50:00.836947 6442 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:50:00.837154 6442 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:50:00.839143 6442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 757b8bfcbeea43c3a7af84de78871b95: Generated new TSK 0
I20260807 22:50:00.839326 6442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:50:00.844199 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 6353
2026-08-07T22:50:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1477 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:50:00 dist-test-slave-rgb1 krb5kdc[6457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:50:00 dist-test-slave-rgb1 krb5kdc[6457](info): set up 2 sockets
Aug 07 22:50:00 dist-test-slave-rgb1 krb5kdc[6457](info): commencing operation
krb5kdc: starting...
W20260807 22:50:02.032967 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:50:02 dist-test-slave-rgb1 krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:50:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:50:02.217362 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43205
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:37313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:43205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:50:02.312949 6470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:50:02.313232 6470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:50:02.313277 6470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:50:02.314515 6470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:50:02.314597 6470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:50:02.314625 6470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:50:02.314656 6470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:50:02.314688 6470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:50:02.316313 6470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:37313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:43205
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:43205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:50:02.316671 6470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:50:02.316941 6470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:50:02.322762 6475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:02.322840 6470 server_base.cc:1034] running on GCE node
W20260807 22:50:02.322892 6478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:02.322767 6476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:02.323459 6470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:50:02.323812 6470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:50:02.325106 6470 hybrid_clock.cc:648] HybridClock initialized: now 1786143002325072 us; error 58 us; skew 500 ppm
Aug 07 22:50:02 dist-test-slave-rgb1 krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143002, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:50:02.333746 6470 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:50:02.334203 6470 webserver.cc:460] Webserver started at http://127.16.98.190:41075/ using document root <none> and password file <none>
I20260807 22:50:02.334478 6470 fs_manager.cc:359] Metadata directory not provided
I20260807 22:50:02.334527 6470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:50:02.334625 6470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:50:02.335721 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "c4d9896aa7134871a5ef0d212588deb4"
format_stamp: "Formatted at 2026-08-07 22:50:02 on dist-test-slave-rgb1"
server_key: "2fc69f89f90edb7b09898274933a2d4f"
server_key_iv: "e5f4972040d0d3f6470ddfe57f8d4478"
server_key_version: "encryptionkey@0"
I20260807 22:50:02.336031 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "c4d9896aa7134871a5ef0d212588deb4"
format_stamp: "Formatted at 2026-08-07 22:50:02 on dist-test-slave-rgb1"
server_key: "2fc69f89f90edb7b09898274933a2d4f"
server_key_iv: "e5f4972040d0d3f6470ddfe57f8d4478"
server_key_version: "encryptionkey@0"
I20260807 22:50:02.337623 6470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260807 22:50:02.338699 6485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:50:02.339028 6470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:50:02.339143 6470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "c4d9896aa7134871a5ef0d212588deb4"
format_stamp: "Formatted at 2026-08-07 22:50:02 on dist-test-slave-rgb1"
server_key: "2fc69f89f90edb7b09898274933a2d4f"
server_key_iv: "e5f4972040d0d3f6470ddfe57f8d4478"
server_key_version: "encryptionkey@0"
I20260807 22:50:02.339207 6470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:50:02.355181 6470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:50:02.357714 6470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:50:02.357847 6470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:50:02.363181 6470 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:43205
I20260807 22:50:02.363358 6541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:43205 every 8 connection(s)
I20260807 22:50:02.363659 6470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:50:02.364956 6542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:50:02.365538 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 6470
I20260807 22:50:02.365715 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:50:02.365962 16778 external_mini_cluster.cc:1428] Setting key 05ecb5a3d324f15123a3a85eb9100765
I20260807 22:50:02.370983 6542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4: Bootstrap starting.
I20260807 22:50:02.372071 6542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4: Neither blocks nor log segments found. Creating new log.
I20260807 22:50:02.372944 6542 log.cc:824] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4: Log is configured to *not* fsync() on all Append() calls
I20260807 22:50:02.374667 6542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4: No bootstrap required, opened a new log
I20260807 22:50:02.377601 6542 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d9896aa7134871a5ef0d212588deb4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43205 } }
I20260807 22:50:02.377954 6542 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:50:02.378013 6542 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d9896aa7134871a5ef0d212588deb4, State: Initialized, Role: FOLLOWER
I20260807 22:50:02.378181 6542 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d9896aa7134871a5ef0d212588deb4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43205 } }
I20260807 22:50:02.378290 6542 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:50:02.378341 6542 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:50:02.378403 6542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:50:02.379851 6542 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d9896aa7134871a5ef0d212588deb4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43205 } }
Aug 07 22:50:02 dist-test-slave-rgb1 krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:50:02.380040 6542 leader_election.cc:302] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4d9896aa7134871a5ef0d212588deb4; no voters:
I20260807 22:50:02.380533 6542 leader_election.cc:288] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:50:02.380606 6547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:50:02.381124 6547 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [term 1 LEADER]: Becoming Leader. State: Replica: c4d9896aa7134871a5ef0d212588deb4, State: Running, Role: LEADER
I20260807 22:50:02.381296 6542 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:50:02.381325 6547 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d9896aa7134871a5ef0d212588deb4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43205 } }
I20260807 22:50:02.382092 6549 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4d9896aa7134871a5ef0d212588deb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d9896aa7134871a5ef0d212588deb4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43205 } } }
I20260807 22:50:02.382093 6548 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4d9896aa7134871a5ef0d212588deb4. Latest consensus state: current_term: 1 leader_uuid: "c4d9896aa7134871a5ef0d212588deb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d9896aa7134871a5ef0d212588deb4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 43205 } } }
I20260807 22:50:02.382215 6549 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:02.382248 6548 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4 [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:02.382624 6556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:50:02.383769 6556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:50:02.386394 6556 catalog_manager.cc:1347] Generated new cluster ID: 823d66770412473f93476ff306e12bf2
I20260807 22:50:02.386533 6556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:50:02.388201 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:02.366627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59473 (local address 127.16.98.190:43205)
0807 22:50:02.366840 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:02.366849 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:02.366910 (+ 61us) server_negotiation.cc:407] Connection header received
0807 22:50:02.368752 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:02.368763 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:02.368837 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:02.368988 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:02.371528 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:02.374273 (+ 2745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:02.375606 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:02.376901 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:02.377275 (+ 374us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:02.381077 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:02.381083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:02.381083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:02.381099 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:02.385212 (+ 4113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:02.386211 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:02.386215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:02.386216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:02.386374 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:02.386777 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:02.386778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:02.386779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:02.387113 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:02.387276 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:02.387692 (+ 416us) server_negotiation.cc:299] Negotiation successful
0807 22:50:02.387819 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":111,"threads_started":1}
I20260807 22:50:02.389551 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:50:02.411201 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:02.393003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34585 (local address 127.16.98.190:43205)
0807 22:50:02.393057 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:02.393059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:02.400902 (+ 7843us) server_negotiation.cc:407] Connection header received
0807 22:50:02.400983 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:02.400985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:02.401001 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:02.401085 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:02.401984 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:02.402778 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:02.403962 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:02.404309 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:02.404373 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:02.406176 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:02.406178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:02.406178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:02.406186 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:02.408679 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:02.409220 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:02.409221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:02.409222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:02.409286 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:02.409561 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:02.409562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:02.409562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:02.409695 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:02.409793 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:02.411015 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0807 22:50:02.411092 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 22:50:02.416162 6556 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:50:02.417284 6556 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:50:02.422887 6556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4d9896aa7134871a5ef0d212588deb4: Generated new TSK 0
I20260807 22:50:02.423197 6556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:50:02.426614 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 6470
W20260807 22:50:02.433102 6455 connection.cc:535] client connection to 127.16.98.190:43205 recv error: Network error: failed to read from TLS socket (remote: 127.16.98.190:43205): Connection reset by peer (error 104)
2026-08-07T22:50:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1581 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:50:02 dist-test-slave-rgb1 krb5kdc[6574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:50:02 dist-test-slave-rgb1 krb5kdc[6574](info): set up 2 sockets
Aug 07 22:50:02 dist-test-slave-rgb1 krb5kdc[6574](info): commencing operation
krb5kdc: starting...
W20260807 22:50:03.571231 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:50:03 dist-test-slave-rgb1 krb5kdc[6574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:50:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:50:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:50:03.722234 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:44239
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:38997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:44239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:50:03.816327 6589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:50:03.816661 6589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:50:03.816721 6589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:50:03.817958 6589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:50:03.818030 6589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:50:03.818058 6589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:50:03.818089 6589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:50:03.818116 6589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:50:03.819687 6589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:38997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:44239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:44239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:50:03.819983 6589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:50:03.820236 6589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:50:03.826254 6597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:03.826280 6595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:03.826256 6594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:03.826550 6589 server_base.cc:1034] running on GCE node
I20260807 22:50:03.826885 6589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:50:03.827222 6589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:50:03.828442 6589 hybrid_clock.cc:648] HybridClock initialized: now 1786143003828420 us; error 49 us; skew 500 ppm
Aug 07 22:50:03 dist-test-slave-rgb1 krb5kdc[6574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143003, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:50:03.831274 6589 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:50:03.831877 6589 webserver.cc:460] Webserver started at http://127.16.98.190:40715/ using document root <none> and password file <none>
I20260807 22:50:03.832147 6589 fs_manager.cc:359] Metadata directory not provided
I20260807 22:50:03.832221 6589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:50:03.832369 6589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:50:03.833581 6589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "4560e3564cb64037a2585f16b6cd1f36"
format_stamp: "Formatted at 2026-08-07 22:50:03 on dist-test-slave-rgb1"
server_key: "b1256164b2781a132734da03d910ae02"
server_key_iv: "2834fdb00da4b2f9d3ceb110ce41ce08"
server_key_version: "encryptionkey@0"
I20260807 22:50:03.833961 6589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "4560e3564cb64037a2585f16b6cd1f36"
format_stamp: "Formatted at 2026-08-07 22:50:03 on dist-test-slave-rgb1"
server_key: "b1256164b2781a132734da03d910ae02"
server_key_iv: "2834fdb00da4b2f9d3ceb110ce41ce08"
server_key_version: "encryptionkey@0"
I20260807 22:50:03.835415 6589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260807 22:50:03.836297 6604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:50:03.836544 6589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:50:03.836716 6589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "4560e3564cb64037a2585f16b6cd1f36"
format_stamp: "Formatted at 2026-08-07 22:50:03 on dist-test-slave-rgb1"
server_key: "b1256164b2781a132734da03d910ae02"
server_key_iv: "2834fdb00da4b2f9d3ceb110ce41ce08"
server_key_version: "encryptionkey@0"
I20260807 22:50:03.836804 6589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:50:03.852236 6589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:50:03.854760 6589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:50:03.854934 6589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:50:03.859648 6589 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:44239
I20260807 22:50:03.859689 6657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:44239 every 8 connection(s)
I20260807 22:50:03.860205 6589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:50:03.861250 6658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:50:03.863718 6658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36: Bootstrap starting.
I20260807 22:50:03.864365 6658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36: Neither blocks nor log segments found. Creating new log.
I20260807 22:50:03.864753 6658 log.cc:824] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36: Log is configured to *not* fsync() on all Append() calls
I20260807 22:50:03.865600 6658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36: No bootstrap required, opened a new log
I20260807 22:50:03.867491 6658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4560e3564cb64037a2585f16b6cd1f36" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44239 } }
I20260807 22:50:03.867645 6658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:50:03.867668 6658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4560e3564cb64037a2585f16b6cd1f36, State: Initialized, Role: FOLLOWER
I20260807 22:50:03.867839 6658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4560e3564cb64037a2585f16b6cd1f36" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44239 } }
I20260807 22:50:03.867909 6658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:50:03.867933 6658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:50:03.867988 6658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:50:03.869109 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 6589
I20260807 22:50:03.869097 6658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4560e3564cb64037a2585f16b6cd1f36" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44239 } }
I20260807 22:50:03.869251 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:50:03.869222 6658 leader_election.cc:302] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4560e3564cb64037a2585f16b6cd1f36; no voters:
I20260807 22:50:03.869416 6658 leader_election.cc:288] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:50:03.869491 16778 external_mini_cluster.cc:1428] Setting key 9b0f4b4e985230390d1ef029f33a8428
I20260807 22:50:03.869483 6662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:50:03.869653 6662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [term 1 LEADER]: Becoming Leader. State: Replica: 4560e3564cb64037a2585f16b6cd1f36, State: Running, Role: LEADER
I20260807 22:50:03.869807 6662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4560e3564cb64037a2585f16b6cd1f36" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44239 } }
I20260807 22:50:03.869810 6658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:50:03.870517 6664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4560e3564cb64037a2585f16b6cd1f36. Latest consensus state: current_term: 1 leader_uuid: "4560e3564cb64037a2585f16b6cd1f36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4560e3564cb64037a2585f16b6cd1f36" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44239 } } }
I20260807 22:50:03.870692 6664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:03.870723 6663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4560e3564cb64037a2585f16b6cd1f36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4560e3564cb64037a2585f16b6cd1f36" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 44239 } } }
I20260807 22:50:03.870785 6663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36 [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:03.871240 6669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:50:03.871817 6669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:50:03.873790 6669 catalog_manager.cc:1347] Generated new cluster ID: 1314960aa9ff4c45a7b26b416b4cf43c
I20260807 22:50:03.873855 6669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:50:03.877504 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:03.870193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60409 (local address 127.16.98.190:44239)
0807 22:50:03.870395 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:03.870406 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:03.870447 (+ 41us) server_negotiation.cc:407] Connection header received
0807 22:50:03.871296 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:03.871323 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:03.871524 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:03.871697 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:03.872093 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:03.873509 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:03.874665 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:03.875291 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:03.875973 (+ 682us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:03.876373 (+ 400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:03.876377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:03.876378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:50:03.876703 (+ 325us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:03.876745 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:50:03.876762 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:03.876864 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:03.877205 (+ 341us) server_negotiation.cc:299] Negotiation successful
0807 22:50:03.877259 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260807 22:50:03.878597 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:50:03.884188 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:03.880245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41795 (local address 127.16.98.190:44239)
0807 22:50:03.880293 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:03.880296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:03.880453 (+ 157us) server_negotiation.cc:407] Connection header received
0807 22:50:03.880534 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:03.880534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:03.880546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:03.880630 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:03.880782 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:03.881619 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:03.882657 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:03.883006 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:03.883075 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:03.883887 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:03.883888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:03.883889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:50:03.883894 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:03.883910 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:50:03.883915 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:03.883978 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:03.884060 (+ 82us) server_negotiation.cc:299] Negotiation successful
0807 22:50:03.884074 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260807 22:50:03.901028 6669 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:50:03.901728 6669 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:50:03.905005 6669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4560e3564cb64037a2585f16b6cd1f36: Generated new TSK 0
I20260807 22:50:03.905216 6669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:50:03.909941 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 6589
W20260807 22:50:03.915860 6572 connection.cc:535] client connection to 127.16.98.190:44239 recv error: Network error: failed to read from TLS socket (remote: 127.16.98.190:44239): Connection reset by peer (error 104)
2026-08-07T22:50:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1481 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:50:03 dist-test-slave-rgb1 krb5kdc[6692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:50:03 dist-test-slave-rgb1 krb5kdc[6692](info): set up 2 sockets
Aug 07 22:50:03 dist-test-slave-rgb1 krb5kdc[6692](info): commencing operation
krb5kdc: starting...
W20260807 22:50:05.084308 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:50:05 dist-test-slave-rgb1 krb5kdc[6692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:50:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:50:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:50:05.238714 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:46781
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:42085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:46781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:50:05.339198 6705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:50:05.339493 6705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:50:05.339524 6705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:50:05.341465 6705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:50:05.341535 6705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:50:05.341552 6705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:50:05.341568 6705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:50:05.341584 6705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:50:05.343670 6705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:42085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:46781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:46781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:50:05.344017 6705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:50:05.344307 6705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:50:05.350419 6713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:05.350522 6711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:05.350575 6710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:05.350821 6705 server_base.cc:1034] running on GCE node
I20260807 22:50:05.351110 6705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:50:05.351470 6705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:50:05.352826 6705 hybrid_clock.cc:648] HybridClock initialized: now 1786143005352812 us; error 48 us; skew 500 ppm
Aug 07 22:50:05 dist-test-slave-rgb1 krb5kdc[6692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143005, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:50:05.356299 6705 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:50:05.356931 6705 webserver.cc:460] Webserver started at http://127.16.98.190:33771/ using document root <none> and password file <none>
I20260807 22:50:05.357218 6705 fs_manager.cc:359] Metadata directory not provided
I20260807 22:50:05.357292 6705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:50:05.357460 6705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:50:05.358667 6705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "e02d21b6955c4cd4bcf77b8980466052"
format_stamp: "Formatted at 2026-08-07 22:50:05 on dist-test-slave-rgb1"
server_key: "a5d9c9569d298b85db304b342d14f803"
server_key_iv: "9749b7c2eeab3bb5b4ec81edbd7c83d1"
server_key_version: "encryptionkey@0"
I20260807 22:50:05.359058 6705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "e02d21b6955c4cd4bcf77b8980466052"
format_stamp: "Formatted at 2026-08-07 22:50:05 on dist-test-slave-rgb1"
server_key: "a5d9c9569d298b85db304b342d14f803"
server_key_iv: "9749b7c2eeab3bb5b4ec81edbd7c83d1"
server_key_version: "encryptionkey@0"
I20260807 22:50:05.360982 6705 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 22:50:05.362562 6721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:50:05.362953 6705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:50:05.363114 6705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "e02d21b6955c4cd4bcf77b8980466052"
format_stamp: "Formatted at 2026-08-07 22:50:05 on dist-test-slave-rgb1"
server_key: "a5d9c9569d298b85db304b342d14f803"
server_key_iv: "9749b7c2eeab3bb5b4ec81edbd7c83d1"
server_key_version: "encryptionkey@0"
I20260807 22:50:05.363232 6705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:50:05.367699 6705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:50:05.371109 6705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:50:05.371310 6705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:50:05.376762 6705 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:46781
I20260807 22:50:05.376871 6773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:46781 every 8 connection(s)
I20260807 22:50:05.377243 6705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:50:05.378355 6774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:50:05.380980 6774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052: Bootstrap starting.
I20260807 22:50:05.381641 6774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052: Neither blocks nor log segments found. Creating new log.
I20260807 22:50:05.381963 6774 log.cc:824] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052: Log is configured to *not* fsync() on all Append() calls
I20260807 22:50:05.382848 6774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052: No bootstrap required, opened a new log
I20260807 22:50:05.385766 6774 raft_consensus.cc:348] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02d21b6955c4cd4bcf77b8980466052" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46781 } }
I20260807 22:50:05.386087 6774 raft_consensus.cc:374] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:50:05.386148 6774 raft_consensus.cc:729] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02d21b6955c4cd4bcf77b8980466052, State: Initialized, Role: FOLLOWER
I20260807 22:50:05.386411 6774 consensus_queue.cc:260] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02d21b6955c4cd4bcf77b8980466052" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46781 } }
I20260807 22:50:05.386538 6774 raft_consensus.cc:388] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:50:05.386584 6774 raft_consensus.cc:482] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:50:05.386643 6774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:50:05.386749 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 6705
I20260807 22:50:05.386898 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:50:05.387156 16778 external_mini_cluster.cc:1428] Setting key 8ff3e37cb703a1aff11a611e073ed229
I20260807 22:50:05.388317 6774 raft_consensus.cc:504] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02d21b6955c4cd4bcf77b8980466052" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46781 } }
I20260807 22:50:05.388473 6774 leader_election.cc:302] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e02d21b6955c4cd4bcf77b8980466052; no voters:
I20260807 22:50:05.388784 6774 leader_election.cc:288] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:50:05.388818 6779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:50:05.388970 6779 raft_consensus.cc:686] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [term 1 LEADER]: Becoming Leader. State: Replica: e02d21b6955c4cd4bcf77b8980466052, State: Running, Role: LEADER
I20260807 22:50:05.389142 6774 sys_catalog.cc:564] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:50:05.389175 6779 consensus_queue.cc:237] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02d21b6955c4cd4bcf77b8980466052" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46781 } }
I20260807 22:50:05.389634 6779 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e02d21b6955c4cd4bcf77b8980466052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02d21b6955c4cd4bcf77b8980466052" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46781 } } }
I20260807 22:50:05.389746 6779 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:05.389956 6779 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e02d21b6955c4cd4bcf77b8980466052. Latest consensus state: current_term: 1 leader_uuid: "e02d21b6955c4cd4bcf77b8980466052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02d21b6955c4cd4bcf77b8980466052" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46781 } } }
I20260807 22:50:05.390025 6779 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052 [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:05.390282 6786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:50:05.390482 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:05.387809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34571 (local address 127.16.98.190:46781)
0807 22:50:05.388123 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:05.388133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:05.388167 (+ 34us) server_negotiation.cc:407] Connection header received
0807 22:50:05.388283 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:05.388289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:05.388318 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:05.388489 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:05.389341 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:05.389345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:05.389346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:50:05.389350 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:05.389401 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:50:05.389413 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:05.389501 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:05.389590 (+ 89us) server_negotiation.cc:299] Negotiation successful
0807 22:50:05.389773 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":136,"threads_started":1}
I20260807 22:50:05.391229 6786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:50:05.391918 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:50:05.393509 6786 catalog_manager.cc:1347] Generated new cluster ID: 69c52e03219844c3a3ad51b09fba6d14
I20260807 22:50:05.393572 6786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:50:05.409770 6786 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:50:05.410027 6786 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:50:05.410450 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:05.409017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34253 (local address 127.16.98.190:46781)
0807 22:50:05.409073 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:05.409076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:05.409227 (+ 151us) server_negotiation.cc:407] Connection header received
0807 22:50:05.409352 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:05.409353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:05.409361 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:05.409418 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:05.410141 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:05.410143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:05.410144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:50:05.410149 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:05.410167 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:50:05.410172 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:05.410227 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:05.410348 (+ 121us) server_negotiation.cc:299] Negotiation successful
0807 22:50:05.410366 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 22:50:05.414175 6786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e02d21b6955c4cd4bcf77b8980466052: Generated new TSK 0
I20260807 22:50:05.414448 6786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:50:05.422067 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 6705
2026-08-07T22:50:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1514 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:50:05 dist-test-slave-rgb1 krb5kdc[6805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:50:05 dist-test-slave-rgb1 krb5kdc[6805](info): set up 2 sockets
Aug 07 22:50:05 dist-test-slave-rgb1 krb5kdc[6805](info): commencing operation
krb5kdc: starting...
W20260807 22:50:06.571651 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:50:06 dist-test-slave-rgb1 krb5kdc[6805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:50:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:50:06.719601 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:32879
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:37011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:32879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:50:06.815295 6820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:50:06.815546 6820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:50:06.815572 6820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:50:06.816798 6820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:50:06.816851 6820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:50:06.816866 6820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:50:06.816879 6820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:50:06.816891 6820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:50:06.818420 6820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:37011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:32879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:32879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:50:06.818645 6820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:50:06.818846 6820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:50:06.824213 6828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:06.824184 6825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:06.824208 6820 server_base.cc:1034] running on GCE node
W20260807 22:50:06.824188 6826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:06.824744 6820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:50:06.825117 6820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:50:06.826313 6820 hybrid_clock.cc:648] HybridClock initialized: now 1786143006826253 us; error 100 us; skew 500 ppm
Aug 07 22:50:06 dist-test-slave-rgb1 krb5kdc[6805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143006, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:50:06.829450 6820 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:50:06.829922 6820 webserver.cc:460] Webserver started at http://127.16.98.190:40897/ using document root <none> and password file <none>
I20260807 22:50:06.830170 6820 fs_manager.cc:359] Metadata directory not provided
I20260807 22:50:06.830221 6820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:50:06.830322 6820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:50:06.831475 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "a1f08dca6ed54e5a86c2bd8cec3136e2"
format_stamp: "Formatted at 2026-08-07 22:50:06 on dist-test-slave-rgb1"
server_key: "c745a7e9997409db2f2c516c0cd550fc"
server_key_iv: "82b7181c4af256d56f5d236ea53ab69d"
server_key_version: "encryptionkey@0"
I20260807 22:50:06.831817 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "a1f08dca6ed54e5a86c2bd8cec3136e2"
format_stamp: "Formatted at 2026-08-07 22:50:06 on dist-test-slave-rgb1"
server_key: "c745a7e9997409db2f2c516c0cd550fc"
server_key_iv: "82b7181c4af256d56f5d236ea53ab69d"
server_key_version: "encryptionkey@0"
I20260807 22:50:06.833434 6820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:50:06.834812 6835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:50:06.835196 6820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:50:06.835295 6820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "a1f08dca6ed54e5a86c2bd8cec3136e2"
format_stamp: "Formatted at 2026-08-07 22:50:06 on dist-test-slave-rgb1"
server_key: "c745a7e9997409db2f2c516c0cd550fc"
server_key_iv: "82b7181c4af256d56f5d236ea53ab69d"
server_key_version: "encryptionkey@0"
I20260807 22:50:06.835419 6820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:50:06.846787 6820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:50:06.849285 6820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:50:06.849469 6820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:50:06.854234 6820 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:32879
I20260807 22:50:06.854277 6889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:32879 every 8 connection(s)
I20260807 22:50:06.854710 6820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:50:06.855791 6890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:50:06.858222 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 6820
I20260807 22:50:06.858386 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:50:06.858603 16778 external_mini_cluster.cc:1428] Setting key ed6f8dc3b35e23f105067b4626ff7ad6
I20260807 22:50:06.859037 6890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2: Bootstrap starting.
I20260807 22:50:06.860181 6890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2: Neither blocks nor log segments found. Creating new log.
I20260807 22:50:06.860605 6890 log.cc:824] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2: Log is configured to *not* fsync() on all Append() calls
I20260807 22:50:06.861675 6890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2: No bootstrap required, opened a new log
I20260807 22:50:06.863991 6890 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f08dca6ed54e5a86c2bd8cec3136e2" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 32879 } }
I20260807 22:50:06.864194 6890 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:50:06.864251 6890 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f08dca6ed54e5a86c2bd8cec3136e2, State: Initialized, Role: FOLLOWER
I20260807 22:50:06.864401 6890 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f08dca6ed54e5a86c2bd8cec3136e2" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 32879 } }
I20260807 22:50:06.864519 6890 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:50:06.864609 6890 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:50:06.864668 6890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:50:06.865355 6890 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f08dca6ed54e5a86c2bd8cec3136e2" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 32879 } }
I20260807 22:50:06.865492 6890 leader_election.cc:302] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1f08dca6ed54e5a86c2bd8cec3136e2; no voters:
I20260807 22:50:06.865736 6890 leader_election.cc:288] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:50:06.865767 6895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:50:06.865887 6895 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [term 1 LEADER]: Becoming Leader. State: Replica: a1f08dca6ed54e5a86c2bd8cec3136e2, State: Running, Role: LEADER
I20260807 22:50:06.865983 6895 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f08dca6ed54e5a86c2bd8cec3136e2" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 32879 } }
I20260807 22:50:06.866074 6890 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:50:06.866387 6895 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1f08dca6ed54e5a86c2bd8cec3136e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f08dca6ed54e5a86c2bd8cec3136e2" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 32879 } } }
I20260807 22:50:06.866458 6895 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:06.866616 6895 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1f08dca6ed54e5a86c2bd8cec3136e2. Latest consensus state: current_term: 1 leader_uuid: "a1f08dca6ed54e5a86c2bd8cec3136e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f08dca6ed54e5a86c2bd8cec3136e2" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 32879 } } }
I20260807 22:50:06.866685 6895 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2 [sys.catalog]: This master's current role is: LEADER
Aug 07 22:50:06 dist-test-slave-rgb1 krb5kdc[6805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:50:06.867368 6899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:50:06.867990 6899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:50:06.871999 6899 catalog_manager.cc:1347] Generated new cluster ID: c661a2f1e3b846e5b0c6084220fc2b40
I20260807 22:50:06.872081 6899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:50:06.873876 6893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:06.859251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46017 (local address 127.16.98.190:32879)
0807 22:50:06.859460 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:06.859471 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:06.859509 (+ 38us) server_negotiation.cc:407] Connection header received
0807 22:50:06.860237 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:06.860247 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:06.860321 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:06.860526 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:06.861507 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:06.862503 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:06.863578 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:06.864010 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:06.864149 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:06.867974 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:06.867979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:06.867979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:06.867994 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:06.871277 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:06.871791 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:06.871794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:06.871794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:06.871904 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:06.872172 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:06.872174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:06.872174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:06.872522 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:06.872735 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:06.872868 (+ 133us) server_negotiation.cc:299] Negotiation successful
0807 22:50:06.873027 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260807 22:50:06.875048 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:50:06.882124 6899 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:50:06.882827 6899 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:50:06.885763 6893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:06.876899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39913 (local address 127.16.98.190:32879)
0807 22:50:06.876938 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:06.876971 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:06.876980 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:50:06.877071 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:06.877072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:06.877082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:06.877138 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:06.877909 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:06.878650 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:06.879801 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:06.880130 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:06.880193 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:06.881939 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:06.881941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:06.881941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:06.881950 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:06.884231 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:06.884759 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:06.884760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:06.884760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:06.884830 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:06.885059 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:06.885060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:06.885060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:06.885196 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:06.885297 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:06.885441 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 22:50:06.885549 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260807 22:50:06.890678 6899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1f08dca6ed54e5a86c2bd8cec3136e2: Generated new TSK 0
I20260807 22:50:06.890904 6899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:50:06.897035 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 6820
2026-08-07T22:50:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1472 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:50:06 dist-test-slave-rgb1 krb5kdc[6922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:50:06 dist-test-slave-rgb1 krb5kdc[6922](info): set up 2 sockets
Aug 07 22:50:06 dist-test-slave-rgb1 krb5kdc[6922](info): commencing operation
krb5kdc: starting...
W20260807 22:50:08.033083 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:50:08 dist-test-slave-rgb1 krb5kdc[6922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:50:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:50:08.189615 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:46687
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:40259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:46687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:50:08.277873 6935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:50:08.278169 6935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:50:08.278213 6935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:50:08.279785 6935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:50:08.279896 6935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:50:08.279929 6935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:50:08.279997 6935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:50:08.280035 6935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:50:08.281973 6935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:40259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:46687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:46687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:50:08.282312 6935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:50:08.282599 6935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:50:08.288801 6940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:08.288793 6941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:08.289368 6935 server_base.cc:1034] running on GCE node
W20260807 22:50:08.288816 6943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:08.289773 6935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:50:08.290231 6935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:50:08.291433 6935 hybrid_clock.cc:648] HybridClock initialized: now 1786143008291408 us; error 45 us; skew 500 ppm
Aug 07 22:50:08 dist-test-slave-rgb1 krb5kdc[6922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143008, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:50:08.298293 6935 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:50:08.298770 6935 webserver.cc:460] Webserver started at http://127.16.98.190:38089/ using document root <none> and password file <none>
I20260807 22:50:08.298982 6935 fs_manager.cc:359] Metadata directory not provided
I20260807 22:50:08.299026 6935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:50:08.299121 6935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:50:08.300205 6935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "dedcfbc485be48cf8ccefb3ce8405714"
format_stamp: "Formatted at 2026-08-07 22:50:08 on dist-test-slave-rgb1"
server_key: "e1d2265c95dbd593137e8573f1d5f6b3"
server_key_iv: "344e442a258d617eb2dd8b58fcf9f8b8"
server_key_version: "encryptionkey@0"
I20260807 22:50:08.300626 6935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "dedcfbc485be48cf8ccefb3ce8405714"
format_stamp: "Formatted at 2026-08-07 22:50:08 on dist-test-slave-rgb1"
server_key: "e1d2265c95dbd593137e8573f1d5f6b3"
server_key_iv: "344e442a258d617eb2dd8b58fcf9f8b8"
server_key_version: "encryptionkey@0"
I20260807 22:50:08.302399 6935 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260807 22:50:08.303750 6951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:50:08.304064 6935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:50:08.304170 6935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "dedcfbc485be48cf8ccefb3ce8405714"
format_stamp: "Formatted at 2026-08-07 22:50:08 on dist-test-slave-rgb1"
server_key: "e1d2265c95dbd593137e8573f1d5f6b3"
server_key_iv: "344e442a258d617eb2dd8b58fcf9f8b8"
server_key_version: "encryptionkey@0"
I20260807 22:50:08.304240 6935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:50:08.330981 6935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:50:08.334004 6935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:50:08.334165 6935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:50:08.339186 6935 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:46687
I20260807 22:50:08.339255 7006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:46687 every 8 connection(s)
I20260807 22:50:08.339627 6935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:50:08.340597 7007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:50:08.343030 7007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714: Bootstrap starting.
I20260807 22:50:08.343673 7007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714: Neither blocks nor log segments found. Creating new log.
I20260807 22:50:08.343982 7007 log.cc:824] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714: Log is configured to *not* fsync() on all Append() calls
I20260807 22:50:08.344815 7007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714: No bootstrap required, opened a new log
I20260807 22:50:08.346506 7007 raft_consensus.cc:348] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dedcfbc485be48cf8ccefb3ce8405714" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46687 } }
I20260807 22:50:08.346679 7007 raft_consensus.cc:374] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:50:08.346730 7007 raft_consensus.cc:729] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dedcfbc485be48cf8ccefb3ce8405714, State: Initialized, Role: FOLLOWER
I20260807 22:50:08.346918 7007 consensus_queue.cc:260] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dedcfbc485be48cf8ccefb3ce8405714" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46687 } }
I20260807 22:50:08.347030 7007 raft_consensus.cc:388] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:50:08.347078 7007 raft_consensus.cc:482] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:50:08.347129 7007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:50:08.347263 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 6935
I20260807 22:50:08.347430 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:50:08.347697 16778 external_mini_cluster.cc:1428] Setting key cbf80c76bff1ffb93954af59dbffdc99
I20260807 22:50:08.348402 7007 raft_consensus.cc:504] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dedcfbc485be48cf8ccefb3ce8405714" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46687 } }
I20260807 22:50:08.348631 7007 leader_election.cc:302] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dedcfbc485be48cf8ccefb3ce8405714; no voters:
I20260807 22:50:08.348812 7007 leader_election.cc:288] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:50:08.348892 7013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:50:08.349086 7007 sys_catalog.cc:564] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:50:08.349143 7013 raft_consensus.cc:686] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [term 1 LEADER]: Becoming Leader. State: Replica: dedcfbc485be48cf8ccefb3ce8405714, State: Running, Role: LEADER
I20260807 22:50:08.349318 7013 consensus_queue.cc:237] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dedcfbc485be48cf8ccefb3ce8405714" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46687 } }
I20260807 22:50:08.349766 7014 sys_catalog.cc:455] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dedcfbc485be48cf8ccefb3ce8405714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dedcfbc485be48cf8ccefb3ce8405714" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46687 } } }
I20260807 22:50:08.349874 7014 sys_catalog.cc:458] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:08.350054 7015 sys_catalog.cc:455] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dedcfbc485be48cf8ccefb3ce8405714. Latest consensus state: current_term: 1 leader_uuid: "dedcfbc485be48cf8ccefb3ce8405714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dedcfbc485be48cf8ccefb3ce8405714" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 46687 } } }
I20260807 22:50:08.350123 7015 sys_catalog.cc:458] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714 [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:08.350363 7021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:50:08.350991 7021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:50:08.352802 7021 catalog_manager.cc:1347] Generated new cluster ID: 8394742bde5a4819a429f778454f1c3a
I20260807 22:50:08.352869 7021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:50:08.355005 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:08.348323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44851 (local address 127.16.98.190:46687)
0807 22:50:08.348603 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:08.348614 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:08.348653 (+ 39us) server_negotiation.cc:407] Connection header received
0807 22:50:08.348784 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:08.348792 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:08.348856 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:08.349013 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:08.349414 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:08.351063 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:08.352376 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:08.353220 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:08.353383 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:08.354307 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:08.354311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:08.354311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:50:08.354527 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:08.354563 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:50:08.354574 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:08.354658 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:08.354742 (+ 84us) server_negotiation.cc:299] Negotiation successful
0807 22:50:08.354811 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":118,"threads_started":1}
I20260807 22:50:08.357918 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:50:08.364315 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:08.359403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33087 (local address 127.16.98.190:46687)
0807 22:50:08.359451 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:08.359453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:08.360861 (+ 1408us) server_negotiation.cc:407] Connection header received
0807 22:50:08.361035 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:08.361036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:08.361057 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:08.361270 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:08.361466 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:08.362053 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:08.362875 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:08.363108 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:08.363195 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:08.364014 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:08.364016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:08.364016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:50:08.364023 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:08.364046 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:50:08.364051 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:08.364114 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:08.364166 (+ 52us) server_negotiation.cc:299] Negotiation successful
0807 22:50:08.364196 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 22:50:08.368876 7021 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:50:08.369510 7021 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:50:08.378762 7021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dedcfbc485be48cf8ccefb3ce8405714: Generated new TSK 0
I20260807 22:50:08.379040 7021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:50:08.386003 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 6935
2026-08-07T22:50:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1488 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:50:08 dist-test-slave-rgb1 krb5kdc[7039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:50:08 dist-test-slave-rgb1 krb5kdc[7039](info): set up 2 sockets
Aug 07 22:50:08 dist-test-slave-rgb1 krb5kdc[7039](info): commencing operation
krb5kdc: starting...
W20260807 22:50:09.523161 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:50:09 dist-test-slave-rgb1 krb5kdc[7039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:50:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:50:09.683285 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:39315
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:39315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:50:09.784091 7052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:50:09.784337 7052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:50:09.784372 7052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:50:09.785665 7052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:50:09.785715 7052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:50:09.785730 7052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:50:09.785743 7052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:50:09.785756 7052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:50:09.787405 7052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:39315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:39315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:50:09.787698 7052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:50:09.787969 7052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:50:09.793344 7060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:09.793469 7058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:09.793351 7057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:09.793627 7052 server_base.cc:1034] running on GCE node
I20260807 22:50:09.794354 7052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:50:09.794907 7052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:50:09.796175 7052 hybrid_clock.cc:648] HybridClock initialized: now 1786143009796143 us; error 40 us; skew 500 ppm
Aug 07 22:50:09 dist-test-slave-rgb1 krb5kdc[7039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143009, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:50:09.799778 7052 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:50:09.800870 7052 webserver.cc:460] Webserver started at http://127.16.98.190:39143/ using document root <none> and password file <none>
I20260807 22:50:09.801177 7052 fs_manager.cc:359] Metadata directory not provided
I20260807 22:50:09.801240 7052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:50:09.801374 7052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:50:09.802953 7052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "dcc342c6a98c4aa1bc6a018c28e8969a"
format_stamp: "Formatted at 2026-08-07 22:50:09 on dist-test-slave-rgb1"
server_key: "f96014d42109adcd806fa315cbf0f14c"
server_key_iv: "abfdd38df31c0e5a1accbcc248943b34"
server_key_version: "encryptionkey@0"
I20260807 22:50:09.803382 7052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "dcc342c6a98c4aa1bc6a018c28e8969a"
format_stamp: "Formatted at 2026-08-07 22:50:09 on dist-test-slave-rgb1"
server_key: "f96014d42109adcd806fa315cbf0f14c"
server_key_iv: "abfdd38df31c0e5a1accbcc248943b34"
server_key_version: "encryptionkey@0"
I20260807 22:50:09.805559 7052 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260807 22:50:09.806862 7067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:50:09.807277 7052 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:50:09.807427 7052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "dcc342c6a98c4aa1bc6a018c28e8969a"
format_stamp: "Formatted at 2026-08-07 22:50:09 on dist-test-slave-rgb1"
server_key: "f96014d42109adcd806fa315cbf0f14c"
server_key_iv: "abfdd38df31c0e5a1accbcc248943b34"
server_key_version: "encryptionkey@0"
I20260807 22:50:09.807538 7052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:50:09.812319 7052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:50:09.815420 7052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:50:09.815603 7052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:50:09.820452 7052 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:39315
I20260807 22:50:09.820628 7122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:39315 every 8 connection(s)
I20260807 22:50:09.820971 7052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:50:09.822109 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:50:09.824548 7123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a: Bootstrap starting.
I20260807 22:50:09.825228 7123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a: Neither blocks nor log segments found. Creating new log.
I20260807 22:50:09.825556 7123 log.cc:824] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a: Log is configured to *not* fsync() on all Append() calls
I20260807 22:50:09.826678 7123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a: No bootstrap required, opened a new log
I20260807 22:50:09.829349 7123 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc342c6a98c4aa1bc6a018c28e8969a" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 39315 } }
I20260807 22:50:09.829530 7123 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:50:09.829553 7123 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc342c6a98c4aa1bc6a018c28e8969a, State: Initialized, Role: FOLLOWER
I20260807 22:50:09.829732 7123 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc342c6a98c4aa1bc6a018c28e8969a" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 39315 } }
I20260807 22:50:09.829830 7123 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:50:09.829870 7123 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:50:09.829948 7123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:50:09.830907 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 7052
I20260807 22:50:09.830953 7123 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc342c6a98c4aa1bc6a018c28e8969a" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 39315 } }
I20260807 22:50:09.831028 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:50:09.831102 7123 leader_election.cc:302] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcc342c6a98c4aa1bc6a018c28e8969a; no voters:
I20260807 22:50:09.831305 7123 leader_election.cc:288] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:50:09.831387 7127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:50:09.831413 16778 external_mini_cluster.cc:1428] Setting key d34a3efe0b2387e7aa45893fe1dadb66
I20260807 22:50:09.831580 7127 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [term 1 LEADER]: Becoming Leader. State: Replica: dcc342c6a98c4aa1bc6a018c28e8969a, State: Running, Role: LEADER
I20260807 22:50:09.831629 7123 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:50:09.831704 7127 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc342c6a98c4aa1bc6a018c28e8969a" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 39315 } }
I20260807 22:50:09.832172 7129 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcc342c6a98c4aa1bc6a018c28e8969a. Latest consensus state: current_term: 1 leader_uuid: "dcc342c6a98c4aa1bc6a018c28e8969a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc342c6a98c4aa1bc6a018c28e8969a" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 39315 } } }
I20260807 22:50:09.832264 7129 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:09.832206 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcc342c6a98c4aa1bc6a018c28e8969a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcc342c6a98c4aa1bc6a018c28e8969a" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 39315 } } }
I20260807 22:50:09.832456 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:09.832530 7138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:50:09.833180 7138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:50:09.834905 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:09.832045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36627 (local address 127.16.98.190:39315)
0807 22:50:09.832695 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:09.832707 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:09.833027 (+ 320us) server_negotiation.cc:407] Connection header received
0807 22:50:09.833173 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:09.833182 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:09.833221 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:09.833517 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:09.834392 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:09.834395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:09.834395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:50:09.834402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:09.834481 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:50:09.834505 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:09.834580 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:09.834686 (+ 106us) server_negotiation.cc:299] Negotiation successful
0807 22:50:09.834732 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":76,"threads_started":1}
I20260807 22:50:09.834996 7138 catalog_manager.cc:1347] Generated new cluster ID: eb9035c4c07c4d80ac5d5f84e1416b5a
I20260807 22:50:09.835039 7138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:50:09.835839 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:50:09.838514 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:09.837327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42769 (local address 127.16.98.190:39315)
0807 22:50:09.837392 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:09.837394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:09.837516 (+ 122us) server_negotiation.cc:407] Connection header received
0807 22:50:09.837580 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:09.837581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:09.837598 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:09.837648 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:09.838270 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:09.838271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:09.838271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 22:50:09.838275 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:09.838289 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 22:50:09.838293 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:09.838341 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:09.838395 (+ 54us) server_negotiation.cc:299] Negotiation successful
0807 22:50:09.838410 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 22:50:09.854514 7138 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:50:09.854732 7138 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:50:09.862787 7138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcc342c6a98c4aa1bc6a018c28e8969a: Generated new TSK 0
I20260807 22:50:09.863370 7138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:50:09.874773 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 7052
2026-08-07T22:50:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1488 ms)
[----------] 18 tests from AuthTokenIssuingTest (26547 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:50:09 dist-test-slave-rgb1 krb5kdc[7156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:50:09 dist-test-slave-rgb1 krb5kdc[7156](info): set up 2 sockets
Aug 07 22:50:09 dist-test-slave-rgb1 krb5kdc[7156](info): commencing operation
krb5kdc: starting...
W20260807 22:50:11.002703 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:50:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:50:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:50:11.149307 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:40441
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:40441,127.16.98.189:45139,127.16.98.188:40899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:50:11.236250 7170 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 22:50:11.236528 7170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:50:11.236560 7170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:50:11.236626 7170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:50:11.237809 7170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:50:11.237895 7170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:50:11.237921 7170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:50:11.237957 7170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:50:11.237986 7170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:50:11.239550 7170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:40441,127.16.98.189:45139,127.16.98.188:40899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:40441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:50:11.239856 7170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:50:11.240181 7170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:50:11.245401 7176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:11.245597 7175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:11.245416 7178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:11.246279 7170 server_base.cc:1034] running on GCE node
I20260807 22:50:11.246546 7170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:50:11.246866 7170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:50:11.248108 7170 hybrid_clock.cc:648] HybridClock initialized: now 1786143011248081 us; error 57 us; skew 500 ppm
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:50:11.251289 7170 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:50:11.252341 7170 webserver.cc:460] Webserver started at http://127.16.98.190:46193/ using document root <none> and password file <none>
I20260807 22:50:11.252683 7170 fs_manager.cc:359] Metadata directory not provided
I20260807 22:50:11.252739 7170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:50:11.252887 7170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:50:11.254060 7170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "2bade1785e6d4b71b0e20a6d58de2327"
format_stamp: "Formatted at 2026-08-07 22:50:11 on dist-test-slave-rgb1"
server_key: "1761da1c22f9121f91f1f01e08d25051"
server_key_iv: "081ab3c3c3471c4cc964d191a2510e7c"
server_key_version: "encryptionkey@0"
I20260807 22:50:11.254460 7170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "2bade1785e6d4b71b0e20a6d58de2327"
format_stamp: "Formatted at 2026-08-07 22:50:11 on dist-test-slave-rgb1"
server_key: "1761da1c22f9121f91f1f01e08d25051"
server_key_iv: "081ab3c3c3471c4cc964d191a2510e7c"
server_key_version: "encryptionkey@0"
I20260807 22:50:11.256011 7170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260807 22:50:11.257054 7185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:50:11.257352 7170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:50:11.257485 7170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "2bade1785e6d4b71b0e20a6d58de2327"
format_stamp: "Formatted at 2026-08-07 22:50:11 on dist-test-slave-rgb1"
server_key: "1761da1c22f9121f91f1f01e08d25051"
server_key_iv: "081ab3c3c3471c4cc964d191a2510e7c"
server_key_version: "encryptionkey@0"
I20260807 22:50:11.257582 7170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:50:11.288759 7170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:50:11.292173 7170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:50:11.292671 7170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:50:11.297499 7170 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:40441
I20260807 22:50:11.297549 7241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:40441 every 8 connection(s)
I20260807 22:50:11.297968 7170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:50:11.299120 7242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:50:11.301177 7242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:50:11.306603 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 7170
I20260807 22:50:11.306751 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:50:11.307003 16778 external_mini_cluster.cc:1428] Setting key 3d4bf03608d33835bbdbda3422f87a7b
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:50:11.313277 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.302107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.16.98.190:40441)
0807 22:50:11.302292 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.302301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.302429 (+ 128us) server_negotiation.cc:407] Connection header received
0807 22:50:11.303015 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.303017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.303036 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.303138 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.303768 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.304815 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.305906 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.306234 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.306328 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.308735 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:11.308745 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:11.308746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:11.308753 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:11.310653 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.311595 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.311598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.311599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.311734 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.312175 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.312176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.312176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.312505 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:11.312655 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:11.312949 (+ 294us) server_negotiation.cc:299] Negotiation successful
0807 22:50:11.313064 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":110,"threads_started":1}
I20260807 22:50:11.313410 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.302047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:40441 (local address 127.0.0.1:53760)
0807 22:50:11.302274 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.302321 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:11.302869 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:11.303186 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:11.303189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:11.303485 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:11.303663 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.303668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.304970 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.304972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.305804 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.305807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.306370 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.306371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.306435 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.307148 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:11.307165 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:11.308647 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:11.310821 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.310843 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.310846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.311473 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.311833 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.311834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.311835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.311980 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.312653 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:11.312661 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:11.312759 (+ 98us) client_negotiation.cc:769] Sending connection context
0807 22:50:11.312887 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 22:50:11.313195 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":76,"thread_start_us":135,"threads_started":1}
I20260807 22:50:11.314517 7242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:50:11.315023 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.314848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:45139 (local address 127.0.0.1:55370)
0807 22:50:11.314917 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.314978 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:45139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 22:50:11.315222 7188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.98.189:45139: connect: Connection refused (error 111)
I20260807 22:50:11.315753 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.315623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:45139 (local address 127.0.0.1:55384)
0807 22:50:11.315682 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.315720 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:45139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260807 22:50:11.315877 7242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.189:45139: Network error: Client connection negotiation failed: client connection to 127.16.98.189:45139: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.16.98.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.189.keytab.
Entry for principal kudu/127.16.98.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.189.keytab.
I20260807 22:50:11.341264 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.189:45139
--webserver_interface=127.16.98.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.189.keytab
--principal=kudu/127.16.98.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:40441,127.16.98.189:45139,127.16.98.188:40899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:50:11.365150 7242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } attempt: 1
I20260807 22:50:11.365753 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.365591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:45139 (local address 127.0.0.1:55394)
0807 22:50:11.365665 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.365717 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:45139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 22:50:11.366312 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.366209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:45139 (local address 127.0.0.1:55398)
0807 22:50:11.366260 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.366291 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:45139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260807 22:50:11.366482 7242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.189:45139: Network error: Client connection negotiation failed: client connection to 127.16.98.189:45139: connect: Connection refused (error 111)
I20260807 22:50:11.434684 7242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } attempt: 2
I20260807 22:50:11.435274 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.435111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:45139 (local address 127.0.0.1:55404)
0807 22:50:11.435186 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.435237 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:45139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 22:50:11.435742 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.435626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:45139 (local address 127.0.0.1:55414)
0807 22:50:11.435684 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.435717 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:45139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 22:50:11.435886 7242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.189:45139: Network error: Client connection negotiation failed: client connection to 127.16.98.189:45139: connect: Connection refused (error 111)
W20260807 22:50:11.438491 7249 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 22:50:11.438746 7249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:50:11.438802 7249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:50:11.438848 7249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:50:11.440040 7249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:50:11.440107 7249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:50:11.440133 7249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:50:11.440164 7249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:50:11.440191 7249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:50:11.441823 7249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:40441,127.16.98.189:45139,127.16.98.188:40899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.189.keytab
--principal=kudu/127.16.98.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.189:45139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.98.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:50:11.442134 7249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:50:11.442392 7249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 22:50:11.447932 7249 server_base.cc:1034] running on GCE node
W20260807 22:50:11.447824 7257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:11.447829 7254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:11.447824 7255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:11.448379 7249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:50:11.448791 7249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:50:11.449957 7249 hybrid_clock.cc:648] HybridClock initialized: now 1786143011449938 us; error 33 us; skew 500 ppm
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:50:11.453059 7249 init.cc:377] Logged in from keytab as kudu/127.16.98.189@KRBTEST.COM (short username kudu)
I20260807 22:50:11.453621 7249 webserver.cc:460] Webserver started at http://127.16.98.189:34725/ using document root <none> and password file <none>
I20260807 22:50:11.453836 7249 fs_manager.cc:359] Metadata directory not provided
I20260807 22:50:11.453883 7249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:50:11.453992 7249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:50:11.455212 7249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/data/instance:
uuid: "f8f8f90c6704405bb67d63957d094631"
format_stamp: "Formatted at 2026-08-07 22:50:11 on dist-test-slave-rgb1"
server_key: "18c6f632ee466c127548b3ded79154bd"
server_key_iv: "799da20cdc22a7e93b3ecb3477f30d0f"
server_key_version: "encryptionkey@0"
I20260807 22:50:11.455581 7249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/wal/instance:
uuid: "f8f8f90c6704405bb67d63957d094631"
format_stamp: "Formatted at 2026-08-07 22:50:11 on dist-test-slave-rgb1"
server_key: "18c6f632ee466c127548b3ded79154bd"
server_key_iv: "799da20cdc22a7e93b3ecb3477f30d0f"
server_key_version: "encryptionkey@0"
I20260807 22:50:11.457489 7249 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 22:50:11.458528 7264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:50:11.458895 7249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:50:11.459034 7249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/wal
uuid: "f8f8f90c6704405bb67d63957d094631"
format_stamp: "Formatted at 2026-08-07 22:50:11 on dist-test-slave-rgb1"
server_key: "18c6f632ee466c127548b3ded79154bd"
server_key_iv: "799da20cdc22a7e93b3ecb3477f30d0f"
server_key_version: "encryptionkey@0"
I20260807 22:50:11.459132 7249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:50:11.487867 7249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:50:11.490453 7249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:50:11.490614 7249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:50:11.495743 7249 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.189:45139
I20260807 22:50:11.495832 7320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.189:45139 every 8 connection(s)
I20260807 22:50:11.496187 7249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/data/info.pb
I20260807 22:50:11.497367 7321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:50:11.498544 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 7249
I20260807 22:50:11.498649 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-1/wal/instance
I20260807 22:50:11.499157 16778 external_mini_cluster.cc:1428] Setting key 32ecdc18c46c46385f6299f4fdbb7e97
I20260807 22:50:11.499166 7321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.189@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.98.188@KRBTEST.COM; defaulting to no policy
I20260807 22:50:11.511348 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.500016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.16.98.190:40441)
0807 22:50:11.500077 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.500080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.500364 (+ 284us) server_negotiation.cc:407] Connection header received
0807 22:50:11.501225 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.501227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.501245 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.501338 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.502227 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.502774 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.504543 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.504834 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.504903 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.507974 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:11.507976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:11.507977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:11.507986 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:11.509832 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.510319 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.510320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.510321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.510382 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.510648 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.510649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.510650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.510779 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:11.510855 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:11.511159 (+ 304us) server_negotiation.cc:299] Negotiation successful
0807 22:50:11.511250 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260807 22:50:11.511587 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.499967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:40441 (local address 127.0.0.1:53774)
0807 22:50:11.500192 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.500259 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:11.501083 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:11.501408 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:11.501412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:11.501823 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:11.502101 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.502107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.502885 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.502887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.504439 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.504442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.504951 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.504952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.505153 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.505804 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:11.505818 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:11.507857 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:11.509934 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.509944 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.509946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.510220 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.510454 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.510455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.510456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.510573 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.510858 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:11.510866 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:11.511016 (+ 150us) client_negotiation.cc:769] Sending connection context
0807 22:50:11.511127 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 22:50:11.511304 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":109,"threads_started":1}
I20260807 22:50:11.512283 7321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.188@KRBTEST.COM" created.
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.189@KRBTEST.COM for kudu/127.16.98.189@KRBTEST.COM
I20260807 22:50:11.523072 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.512647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:45139 (local address 127.0.0.1:55430)
0807 22:50:11.512681 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.512693 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:11.512734 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:11.513180 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:11.513182 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:11.513350 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:11.513417 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.513419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.514733 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.514734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.515311 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.515313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.516066 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.516067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.516131 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.516737 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:11.516743 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:11.518315 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:11.521115 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.521116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.521116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.521395 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.522054 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.522055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.522056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.522124 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.522735 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:11.522736 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:11.522792 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 22:50:11.522870 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 22:50:11.522968 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260807 22:50:11.523265 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.512730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55430 (local address 127.16.98.189:45139)
0807 22:50:11.512938 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.512942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.513017 (+ 75us) server_negotiation.cc:407] Connection header received
0807 22:50:11.513037 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.513039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.513072 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.513162 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.513507 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.514619 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.515399 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.515916 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.516000 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.518408 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:11.518418 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:11.518418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:11.518426 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:11.520977 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.521504 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.521509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.521509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.521590 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.522221 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.522222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.522223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.522605 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:11.522726 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:11.522951 (+ 225us) server_negotiation.cc:299] Negotiation successful
0807 22:50:11.523053 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":120,"threads_started":1}
I20260807 22:50:11.524055 7321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:50:11.524513 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.524336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:40899 (local address 127.0.0.1:34178)
0807 22:50:11.524430 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.524489 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260807 22:50:11.524766 7266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
I20260807 22:50:11.525468 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.525326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:40899 (local address 127.0.0.1:34192)
0807 22:50:11.525404 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.525436 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 22:50:11.525627 7321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:40899: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
I20260807 22:50:11.527068 7242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } attempt: 3
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for kudu/127.16.98.189@KRBTEST.COM
I20260807 22:50:11.534987 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.527382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:45139 (local address 127.0.0.1:55438)
0807 22:50:11.527441 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.527453 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:11.527519 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:11.527657 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:11.527658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:11.527806 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:11.527876 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.527878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.528792 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.528793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.529395 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.529397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.529894 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.529896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.529991 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.530440 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:11.530444 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:11.531961 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:11.533804 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.533805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.533805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.534023 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.534247 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.534248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.534248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.534313 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.534583 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:11.534584 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:11.534636 (+ 52us) client_negotiation.cc:769] Sending connection context
0807 22:50:11.534757 (+ 121us) client_negotiation.cc:240] Negotiation successful
0807 22:50:11.534886 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260807 22:50:11.534987 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.527441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.16.98.189:45139)
0807 22:50:11.527482 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.527484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.527494 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:50:11.527567 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.527568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.527577 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.527635 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.528047 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.528702 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.529488 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.529759 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.529866 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.532059 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:11.532061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:11.532061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:11.532068 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:11.533696 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.534106 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.534108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.534108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.534165 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.534382 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.534383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.534383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.534511 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:11.534591 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:11.534787 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 22:50:11.534881 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260807 22:50:11.535698 7242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:50:11.536109 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.535963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:40899 (local address 127.0.0.1:34206)
0807 22:50:11.536025 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.536064 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.188.keytab.
Entry for principal kudu/127.16.98.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.188.keytab.
I20260807 22:50:11.536624 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.536462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:40899 (local address 127.0.0.1:34212)
0807 22:50:11.536520 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.536552 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260807 22:50:11.536751 7242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:40899: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
I20260807 22:50:11.537218 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.188:40899
--webserver_interface=127.16.98.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.188.keytab
--principal=kudu/127.16.98.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:40441,127.16.98.189:45139,127.16.98.188:40899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:50:11.567945 7242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } attempt: 1
I20260807 22:50:11.568490 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.568340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:40899 (local address 127.0.0.1:34226)
0807 22:50:11.568413 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.568459 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260807 22:50:11.568970 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.568817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:40899 (local address 127.0.0.1:34238)
0807 22:50:11.568891 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.568938 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260807 22:50:11.569145 7242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:40899: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
I20260807 22:50:11.574887 7321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } attempt: 1
I20260807 22:50:11.575407 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.575250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:40899 (local address 127.0.0.1:34244)
0807 22:50:11.575319 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.575367 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260807 22:50:11.576102 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.575828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:40899 (local address 127.0.0.1:34260)
0807 22:50:11.576021 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.576063 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260807 22:50:11.576265 7321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:40899: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
W20260807 22:50:11.625802 7329 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 22:50:11.626123 7329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:50:11.626157 7329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:50:11.626190 7329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:50:11.628055 7329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:50:11.628127 7329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:50:11.628154 7329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:50:11.628173 7329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:50:11.628190 7329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:50:11.630330 7329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:40441,127.16.98.189:45139,127.16.98.188:40899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.188.keytab
--principal=kudu/127.16.98.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.188:40899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.98.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:50:11.630563 7329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:50:11.630772 7329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:50:11.635905 7334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:11.635962 7337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:11.635944 7335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:11.636178 7329 server_base.cc:1034] running on GCE node
I20260807 22:50:11.636401 7329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:50:11.636724 7329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:50:11.637900 7329 hybrid_clock.cc:648] HybridClock initialized: now 1786143011637873 us; error 43 us; skew 500 ppm
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:50:11.640677 7329 init.cc:377] Logged in from keytab as kudu/127.16.98.188@KRBTEST.COM (short username kudu)
I20260807 22:50:11.641211 7329 webserver.cc:460] Webserver started at http://127.16.98.188:40731/ using document root <none> and password file <none>
I20260807 22:50:11.641448 7329 fs_manager.cc:359] Metadata directory not provided
I20260807 22:50:11.641517 7329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:50:11.641678 7329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:50:11.642834 7329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/data/instance:
uuid: "d26aee0781474840a561b07957574563"
format_stamp: "Formatted at 2026-08-07 22:50:11 on dist-test-slave-rgb1"
server_key: "5a68ee4d24528047a7f043db7d07a6b5"
server_key_iv: "007b0beb5007e132cc2d4250b4d610a4"
server_key_version: "encryptionkey@0"
I20260807 22:50:11.643218 7329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/wal/instance:
uuid: "d26aee0781474840a561b07957574563"
format_stamp: "Formatted at 2026-08-07 22:50:11 on dist-test-slave-rgb1"
server_key: "5a68ee4d24528047a7f043db7d07a6b5"
server_key_iv: "007b0beb5007e132cc2d4250b4d610a4"
server_key_version: "encryptionkey@0"
I20260807 22:50:11.644333 7242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } attempt: 2
I20260807 22:50:11.644430 7321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } attempt: 2
I20260807 22:50:11.644798 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.644638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:40899 (local address 127.0.0.1:34268)
0807 22:50:11.644711 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.644755 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260807 22:50:11.644793 7329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260807 22:50:11.644902 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.644755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:40899 (local address 127.0.0.1:34276)
0807 22:50:11.644829 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.644868 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 22:50:11.645201 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.645101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:40899 (local address 127.0.0.1:34278)
0807 22:50:11.645149 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.645174 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260807 22:50:11.645295 7242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:40899: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
I20260807 22:50:11.645421 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.645299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:40899 (local address 127.0.0.1:34290)
0807 22:50:11.645365 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.645395 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260807 22:50:11.645542 7321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:40899: Network error: Client connection negotiation failed: client connection to 127.16.98.188:40899: connect: Connection refused (error 111)
I20260807 22:50:11.645905 7344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:50:11.646132 7329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:50:11.646205 7329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/wal
uuid: "d26aee0781474840a561b07957574563"
format_stamp: "Formatted at 2026-08-07 22:50:11 on dist-test-slave-rgb1"
server_key: "5a68ee4d24528047a7f043db7d07a6b5"
server_key_iv: "007b0beb5007e132cc2d4250b4d610a4"
server_key_version: "encryptionkey@0"
I20260807 22:50:11.646266 7329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:50:11.661078 7329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:50:11.664011 7329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:50:11.664194 7329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:50:11.668906 7329 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.188:40899
I20260807 22:50:11.668968 7398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.188:40899 every 8 connection(s)
I20260807 22:50:11.669399 7329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/data/info.pb
I20260807 22:50:11.670668 7399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:50:11.672278 7399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:50:11.673882 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 7329
I20260807 22:50:11.674081 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786142716408518-16778-0/minicluster-data/master-2/wal/instance
I20260807 22:50:11.674300 16778 external_mini_cluster.cc:1428] Setting key 7042c4670e78aa6d8dda69f1572d8c9f
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.188@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.188@KRBTEST.COM
I20260807 22:50:11.684476 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.673110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53776 (local address 127.16.98.190:40441)
0807 22:50:11.673243 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.673246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.673432 (+ 186us) server_negotiation.cc:407] Connection header received
0807 22:50:11.674001 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.674002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.674014 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.674069 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.674868 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.675418 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.676865 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.677103 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.677175 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.680149 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:11.680151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:11.680152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:11.680162 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:11.682426 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.683127 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.683129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.683129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.683221 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.683664 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.683665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.683665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.683818 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:11.683905 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:11.684244 (+ 339us) server_negotiation.cc:299] Negotiation successful
0807 22:50:11.684359 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260807 22:50:11.685089 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.673023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:40441 (local address 127.0.0.1:53776)
0807 22:50:11.673296 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.673335 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:11.673881 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:11.674156 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:11.674178 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:11.674530 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:11.674756 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.674762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.675555 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.675556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.676751 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.676753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.677221 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.677222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.677372 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.678126 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:11.678136 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:11.680020 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:11.682527 (+ 2507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.682537 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.682539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.683002 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.683324 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.683325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.683326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.683587 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.683909 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:11.683919 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:11.684084 (+ 165us) client_negotiation.cc:769] Sending connection context
0807 22:50:11.684206 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 22:50:11.684441 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":104,"threads_started":1}
I20260807 22:50:11.685853 7399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:50:11.686437 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.674778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34306 (local address 127.16.98.188:40899)
0807 22:50:11.674981 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.674990 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.675712 (+ 722us) server_negotiation.cc:407] Connection header received
0807 22:50:11.675740 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.675743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.675782 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.675868 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.676594 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.677605 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.678540 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.678966 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.679059 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.682487 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:11.682500 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:11.682501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:11.682509 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:11.684686 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.685121 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.685124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.685124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.685212 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.685522 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.685523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.685523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.685751 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:11.685887 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:11.686061 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 22:50:11.686167 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":172,"thread_start_us":120,"threads_started":1}
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.188@KRBTEST.COM for kudu/127.16.98.189@KRBTEST.COM
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:50:11.694118 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.686237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:45139 (local address 127.0.0.1:55454)
0807 22:50:11.686282 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.686291 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:11.686333 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:11.686674 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:11.686675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:11.686822 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:11.686867 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.686868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.687809 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.687810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.688447 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.688449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.689102 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.689107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.689173 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.689668 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:11.689673 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:11.691057 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:11.692857 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.692859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.692859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.693058 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.693401 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.693402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.693402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.693452 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.693817 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:11.693818 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:11.693862 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 22:50:11.693919 (+ 57us) client_negotiation.cc:240] Negotiation successful
0807 22:50:11.693999 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260807 22:50:11.694183 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.686331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.16.98.189:45139)
0807 22:50:11.686380 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.686382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.686391 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:50:11.686421 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.686422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.686434 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.686510 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.686974 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.687720 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.688544 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.688969 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.689034 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.691157 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:11.691158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:11.691159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:11.691167 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:11.692772 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.693250 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.693251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.693251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.693316 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.693570 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.693571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.693571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.693738 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:11.693818 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:11.693973 (+ 155us) server_negotiation.cc:299] Negotiation successful
0807 22:50:11.694059 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260807 22:50:11.694582 7399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:50:11.696587 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.687258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.16.98.190:40441)
0807 22:50:11.687304 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.687306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.687314 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:50:11.687342 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.687343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.687357 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.687489 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.688142 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.688818 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.689915 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.690123 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.690263 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.693079 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:11.693081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:11.693081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:11.693128 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:11.695381 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.695738 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.695739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.695740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.695801 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.696008 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.696009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.696010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.696131 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:11.696314 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:11.696352 (+ 38us) server_negotiation.cc:299] Negotiation successful
0807 22:50:11.696435 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.188@KRBTEST.COM for kudu/127.16.98.188@KRBTEST.COM
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.189@KRBTEST.COM
I20260807 22:50:11.702993 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.695022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34310 (local address 127.16.98.188:40899)
0807 22:50:11.695067 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.695069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.695082 (+ 13us) server_negotiation.cc:407] Connection header received
0807 22:50:11.695182 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.695182 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.695192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.695276 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.695571 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.696110 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.697157 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.697516 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.697590 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.699569 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:11.699571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:11.699571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:11.699578 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:11.701341 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.701976 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.701977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.701977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.702042 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.702322 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.702323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.702324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.702459 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:11.702539 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:11.702737 (+ 198us) server_negotiation.cc:299] Negotiation successful
0807 22:50:11.702837 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":26}
I20260807 22:50:11.703087 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.694957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:40899 (local address 127.0.0.1:34310)
0807 22:50:11.695041 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.695053 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:11.695099 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:11.695278 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:11.695279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:11.695432 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:11.695464 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.695465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.696220 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.696221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.697072 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.697075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.697589 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.697589 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.697645 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.698182 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:11.698187 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:11.699477 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:11.701514 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.701515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.701516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.701805 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.702125 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.702126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.702126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.702208 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.702544 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:11.702545 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:11.702605 (+ 60us) client_negotiation.cc:769] Sending connection context
0807 22:50:11.702715 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 22:50:11.702918 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":60}
I20260807 22:50:11.705472 7399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563: Bootstrap starting.
I20260807 22:50:11.706300 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.697274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55456 (local address 127.16.98.189:45139)
0807 22:50:11.697319 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.697321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.697365 (+ 44us) server_negotiation.cc:407] Connection header received
0807 22:50:11.697415 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.697416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.697427 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.697494 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.697958 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.698522 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.699337 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.699657 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.699711 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.702909 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:11.702911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:11.702911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:11.702919 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:11.704671 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.705147 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.705148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.705148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.705205 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.705462 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.705462 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.705463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.705580 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:11.705763 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:11.706051 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 22:50:11.706174 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260807 22:50:11.706662 7399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563: Neither blocks nor log segments found. Creating new log.
I20260807 22:50:11.707049 7399 log.cc:824] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563: Log is configured to *not* fsync() on all Append() calls
I20260807 22:50:11.708055 7399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563: No bootstrap required, opened a new log
I20260807 22:50:11.708227 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:50:11.709367 7213 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2bade1785e6d4b71b0e20a6d58de2327"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53786
I20260807 22:50:11.709980 7399 raft_consensus.cc:348] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } }
I20260807 22:50:11.710140 7399 raft_consensus.cc:729] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26aee0781474840a561b07957574563, State: Initialized, Role: FOLLOWER
I20260807 22:50:11.710249 7399 consensus_queue.cc:260] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } }
I20260807 22:50:11.710599 7405 sys_catalog.cc:455] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } } }
I20260807 22:50:11.710688 7399 sys_catalog.cc:564] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:50:11.710704 7405 sys_catalog.cc:458] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 22:50:11.713011 7417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:50:11.713081 7417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:50:11.713318 7329 master_runner.cc:186] Error getting master registration for 127.16.98.190:40441: OK, instance_id { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" instance_seqno: 1786143011296117 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 22:50:11.716776 7329 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 22:50:11.736685 7321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } attempt: 3
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.189@KRBTEST.COM for kudu/127.16.98.188@KRBTEST.COM
I20260807 22:50:11.744455 7242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } attempt: 3
I20260807 22:50:11.746938 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.737144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:40899 (local address 127.0.0.1:34326)
0807 22:50:11.737246 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.737261 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:11.737315 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:11.737506 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:11.737507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:11.737730 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:11.737791 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.737793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.738546 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.738547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.739530 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.739533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.739970 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.739971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.740037 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.740626 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:11.740632 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:11.742596 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:11.745620 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.745621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.745622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.745892 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.746134 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.746135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.746135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.746194 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.746580 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:11.746581 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:11.746638 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 22:50:11.746714 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 22:50:11.746827 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260807 22:50:11.746959 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.737279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.16.98.188:40899)
0807 22:50:11.737359 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.737362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.737373 (+ 11us) server_negotiation.cc:407] Connection header received
0807 22:50:11.737404 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.737405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.737416 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.737498 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.737904 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.738451 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.739659 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.739889 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.739974 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.742947 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:11.742949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:11.742950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:11.742961 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:11.745033 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.745986 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.745988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.745988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.746062 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.746264 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.746265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.746265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.746490 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:11.746571 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:11.746758 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 22:50:11.746858 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 22:50:11.748929 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631: Bootstrap starting.
I20260807 22:50:11.749728 7321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631: Neither blocks nor log segments found. Creating new log.
I20260807 22:50:11.750059 7321 log.cc:824] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631: Log is configured to *not* fsync() on all Append() calls
Aug 07 22:50:11 dist-test-slave-rgb1 krb5kdc[7156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for kudu/127.16.98.188@KRBTEST.COM
I20260807 22:50:11.751783 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631: No bootstrap required, opened a new log
I20260807 22:50:11.754736 7321 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } }
I20260807 22:50:11.754910 7321 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f8f90c6704405bb67d63957d094631, State: Initialized, Role: FOLLOWER
I20260807 22:50:11.755012 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.744853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:40899 (local address 127.0.0.1:34334)
0807 22:50:11.744912 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:11.744924 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:11.744962 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:11.745842 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:11.745844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:11.746008 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:11.746057 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.746058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.747369 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.747371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.748026 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:11.748028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.748606 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.748607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:11.748675 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.749172 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:11.749179 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:11.751332 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:11.753545 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.753546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.753546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.753798 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.754137 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:11.754138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:11.754138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:11.754211 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:11.754540 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:11.754541 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:11.754649 (+ 108us) client_negotiation.cc:769] Sending connection context
0807 22:50:11.754738 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 22:50:11.754898 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260807 22:50:11.755038 7321 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } }
I20260807 22:50:11.755494 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.745431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.16.98.188:40899)
0807 22:50:11.745665 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.745668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.745678 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:50:11.745702 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.745703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.745726 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.745798 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.746145 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.747284 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.748124 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.748489 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.748622 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.751431 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:11.751433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:11.751433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:11.751441 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:11.753420 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.753914 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.753915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.753915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.754030 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.754289 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.754290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.754290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.754456 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:11.754552 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:11.754784 (+ 232us) server_negotiation.cc:299] Negotiation successful
0807 22:50:11.755299 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":132,"threads_started":1}
I20260807 22:50:11.755630 7321 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:50:11.755653 7421 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } } }
I20260807 22:50:11.755847 7421 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:50:11.757513 7242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327: Bootstrap starting.
W20260807 22:50:11.757974 7432 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:50:11.758030 7432 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:50:11.758417 7242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327: Neither blocks nor log segments found. Creating new log.
I20260807 22:50:11.758757 7242 log.cc:824] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327: Log is configured to *not* fsync() on all Append() calls
I20260807 22:50:11.758078 7249 master_runner.cc:186] Error getting master registration for 127.16.98.190:40441: OK, instance_id { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" instance_seqno: 1786143011296117 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 22:50:11.761226 7249 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 22:50:11.769544 7242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327: No bootstrap required, opened a new log
I20260807 22:50:11.771248 7242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } }
I20260807 22:50:11.771447 7242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bade1785e6d4b71b0e20a6d58de2327, State: Initialized, Role: FOLLOWER
I20260807 22:50:11.771539 7242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } }
I20260807 22:50:11.771932 7242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:50:11.771965 7213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 22:50:11.772099 7213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:50:11.773106 7213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } }
I20260807 22:50:11.773394 7213 leader_election.cc:288] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [CANDIDATE]: Term 1 election: Requested vote from peers f8f8f90c6704405bb67d63957d094631 (127.16.98.189:45139), d26aee0781474840a561b07957574563 (127.16.98.188:40899)
I20260807 22:50:11.774083 7293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2bade1785e6d4b71b0e20a6d58de2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f8f8f90c6704405bb67d63957d094631"
I20260807 22:50:11.774221 7293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:50:11.775178 7293 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bade1785e6d4b71b0e20a6d58de2327 in term 1.
I20260807 22:50:11.775465 7371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2bade1785e6d4b71b0e20a6d58de2327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d26aee0781474840a561b07957574563"
I20260807 22:50:11.775580 7371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:50:11.776930 7371 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bade1785e6d4b71b0e20a6d58de2327 in term 1.
I20260807 22:50:11.778402 7188 leader_election.cc:302] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bade1785e6d4b71b0e20a6d58de2327, f8f8f90c6704405bb67d63957d094631; no voters:
I20260807 22:50:11.780740 7435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } } }
I20260807 22:50:11.780860 7435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:50:11.780926 7435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:50:11.780982 7435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [term 1 LEADER]: Becoming Leader. State: Replica: 2bade1785e6d4b71b0e20a6d58de2327, State: Running, Role: LEADER
I20260807 22:50:11.781071 7435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } }
I20260807 22:50:11.781689 7435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bade1785e6d4b71b0e20a6d58de2327. Latest consensus state: current_term: 1 leader_uuid: "2bade1785e6d4b71b0e20a6d58de2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } } }
I20260807 22:50:11.781786 7170 master_runner.cc:263] No leader master found from master 2bade1785e6d4b71b0e20a6d58de2327
I20260807 22:50:11.781840 7170 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 22:50:11.781795 7435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [sys.catalog]: This master's current role is: LEADER
W20260807 22:50:11.774402 7447 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:50:11.782336 7447 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:50:11.790086 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.780897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.16.98.190:40441)
0807 22:50:11.780949 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.780951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.781929 (+ 978us) server_negotiation.cc:407] Connection header received
0807 22:50:11.782047 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.782048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.782060 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.782093 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.783040 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.783800 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.784933 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.785244 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.785287 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.786983 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:11.786985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:11.786985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:11.786992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:11.788781 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.789214 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.789215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.789216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.789273 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.789470 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.789471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.789471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.789603 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:11.789675 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:11.789893 (+ 218us) server_negotiation.cc:299] Negotiation successful
0807 22:50:11.789974 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 22:50:11.793340 7457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260807 22:50:11.793738 7191 tablet.cc:2369] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:50:11.794019 7457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:50:11.802983 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.781560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.16.98.188:40899)
0807 22:50:11.781614 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.781616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.789972 (+ 8356us) server_negotiation.cc:407] Connection header received
0807 22:50:11.790086 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.790088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.790103 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.790138 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.792703 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.793445 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.794948 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.795281 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.795319 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.798459 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:11.798461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:11.798461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:11.798470 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:11.800347 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.801176 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.801178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.801179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.801246 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.801756 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.801757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.801757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.801931 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:11.802029 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:11.802721 (+ 692us) server_negotiation.cc:299] Negotiation successful
0807 22:50:11.802864 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260807 22:50:11.804207 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:11.781162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.16.98.189:45139)
0807 22:50:11.781238 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.781240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.788987 (+ 7747us) server_negotiation.cc:407] Connection header received
0807 22:50:11.789107 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.789109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.789125 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.789159 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.790919 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.791739 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.792970 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:11.793277 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:11.793331 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:11.795277 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:11.795279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:11.795279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:11.795287 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:11.797396 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.801478 (+ 4082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.801479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.801480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.801544 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:11.802248 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:11.802250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:11.802250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:11.802405 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:11.802493 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:11.803996 (+ 1503us) server_negotiation.cc:299] Negotiation successful
0807 22:50:11.804100 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 22:50:11.806033 7371 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563 [term 1 FOLLOWER]: Refusing update from remote peer 2bade1785e6d4b71b0e20a6d58de2327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:50:11.806354 7293 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631 [term 1 FOLLOWER]: Refusing update from remote peer 2bade1785e6d4b71b0e20a6d58de2327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:50:11.806717 7435 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:50:11.807132 7435 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:50:11.808722 7421 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bade1785e6d4b71b0e20a6d58de2327. Latest consensus state: current_term: 1 leader_uuid: "2bade1785e6d4b71b0e20a6d58de2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } } }
I20260807 22:50:11.808825 7421 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:50:11.809916 7405 sys_catalog.cc:455] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bade1785e6d4b71b0e20a6d58de2327. Latest consensus state: current_term: 1 leader_uuid: "2bade1785e6d4b71b0e20a6d58de2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } } }
I20260807 22:50:11.810029 7405 sys_catalog.cc:458] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:50:11.813102 7435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bade1785e6d4b71b0e20a6d58de2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } } }
I20260807 22:50:11.813215 7435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:11.813284 7459 mvcc.cc:205] Tried to move back new op lower bound from 7316041776353185792 to 7316041776256266240. Current Snapshot: MvccSnapshot[applied={T|T < 7316041776353185792}]
I20260807 22:50:11.813746 7405 sys_catalog.cc:455] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bade1785e6d4b71b0e20a6d58de2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } } }
I20260807 22:50:11.813839 7405 sys_catalog.cc:458] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:50:11.814138 7462 mvcc.cc:205] Tried to move back new op lower bound from 7316041776353185792 to 7316041776256266240. Current Snapshot: MvccSnapshot[applied={T|T < 7316041776353185792}]
I20260807 22:50:11.814536 7453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bade1785e6d4b71b0e20a6d58de2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } } }
I20260807 22:50:11.814638 7453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327 [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:11.814970 7421 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bade1785e6d4b71b0e20a6d58de2327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bade1785e6d4b71b0e20a6d58de2327" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 40441 } } peers { permanent_uuid: "f8f8f90c6704405bb67d63957d094631" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 45139 } } peers { permanent_uuid: "d26aee0781474840a561b07957574563" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 40899 } } }
I20260807 22:50:11.815078 7421 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:50:11.815749 7460 mvcc.cc:205] Tried to move back new op lower bound from 7316041776353185792 to 7316041776256266240. Current Snapshot: MvccSnapshot[applied={T|T < 7316041776353185792}]
I20260807 22:50:11.821102 7457 catalog_manager.cc:1347] Generated new cluster ID: 72650c22753d442a8d5043cc73cd0c83
I20260807 22:50:11.821187 7457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:50:11.841787 7457 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:50:11.842562 7457 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:50:11.851379 7457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bade1785e6d4b71b0e20a6d58de2327: Generated new TSK 0
I20260807 22:50:11.851624 7457 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 22:50:11.881304 7326 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:11.879750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55464 (local address 127.16.98.189:45139)
0807 22:50:11.879790 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.879792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.880539 (+ 747us) server_negotiation.cc:407] Connection header received
0807 22:50:11.880680 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.880682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.880693 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.880723 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.881232 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55464: BlockingRecv error: recv got EOF from 127.0.0.1:55464 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260807 22:50:11.882076 7245 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:11.879514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53792 (local address 127.16.98.190:40441)
0807 22:50:11.879564 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.879566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.881558 (+ 1992us) server_negotiation.cc:407] Connection header received
0807 22:50:11.881652 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.881654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.881665 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.881694 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.882008 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53792: BlockingRecv error: recv got EOF from 127.0.0.1:53792 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260807 22:50:11.882814 7419 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:11.880241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.16.98.188:40899)
0807 22:50:11.880283 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.880285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.882307 (+ 2022us) server_negotiation.cc:407] Connection header received
0807 22:50:11.882407 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.882408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.882420 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.882450 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.882755 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34360: BlockingRecv error: recv got EOF from 127.0.0.1:34360 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260807 22:50:11.885267 7326 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:11.884421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.16.98.189:45139)
0807 22:50:11.884461 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.884463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.884733 (+ 270us) server_negotiation.cc:407] Connection header received
0807 22:50:11.884840 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.884841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.884861 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.884890 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.885217 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55480: BlockingRecv error: recv got EOF from 127.0.0.1:55480 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260807 22:50:11.888459 7326 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:11.887579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55492 (local address 127.16.98.189:45139)
0807 22:50:11.887615 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.887616 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.887888 (+ 272us) server_negotiation.cc:407] Connection header received
0807 22:50:11.888010 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.888011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.888024 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.888053 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.888406 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55492: BlockingRecv error: recv got EOF from 127.0.0.1:55492 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260807 22:50:11.892989 7326 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:11.891878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55496 (local address 127.16.98.189:45139)
0807 22:50:11.891925 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.891928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.892241 (+ 313us) server_negotiation.cc:407] Connection header received
0807 22:50:11.892363 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.892365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.892379 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.892414 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.892923 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55496: BlockingRecv error: recv got EOF from 127.0.0.1:55496 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260807 22:50:11.899386 7326 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:11.898328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.16.98.189:45139)
0807 22:50:11.898375 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.898377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.898721 (+ 344us) server_negotiation.cc:407] Connection header received
0807 22:50:11.898844 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.898845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.898858 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.898892 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.899319 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55504: BlockingRecv error: recv got EOF from 127.0.0.1:55504 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260807 22:50:11.910053 7326 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:11.908965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.16.98.189:45139)
0807 22:50:11.909020 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.909023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.909344 (+ 321us) server_negotiation.cc:407] Connection header received
0807 22:50:11.909469 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.909470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.909484 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.909536 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.909982 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55520: BlockingRecv error: recv got EOF from 127.0.0.1:55520 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260807 22:50:11.928483 7326 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:11.927608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.16.98.189:45139)
0807 22:50:11.927658 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.927660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.927795 (+ 135us) server_negotiation.cc:407] Connection header received
0807 22:50:11.927876 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.927878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.927892 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.927966 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.928418 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55536: BlockingRecv error: recv got EOF from 127.0.0.1:55536 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260807 22:50:11.962548 7326 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:11.961714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.16.98.189:45139)
0807 22:50:11.961763 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:11.961765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:11.961928 (+ 163us) server_negotiation.cc:407] Connection header received
0807 22:50:11.961978 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:11.961979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:11.962002 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:11.962076 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:11.962478 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55548: BlockingRecv error: recv got EOF from 127.0.0.1:55548 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260807 22:50:12.028707 7326 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:12.027864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.16.98.189:45139)
0807 22:50:12.027984 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:12.027987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:12.027997 (+ 10us) server_negotiation.cc:407] Connection header received
0807 22:50:12.028086 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:12.028087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:12.028098 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:12.028166 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:12.028602 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55560: BlockingRecv error: recv got EOF from 127.0.0.1:55560 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260807 22:50:12.158843 7326 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:12.158131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55574 (local address 127.16.98.189:45139)
0807 22:50:12.158205 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:12.158208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:12.158265 (+ 57us) server_negotiation.cc:407] Connection header received
0807 22:50:12.158349 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:12.158350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:12.158361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:12.158411 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:12.158789 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55574: BlockingRecv error: recv got EOF from 127.0.0.1:55574 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260807 22:50:12.417677 7326 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:12.416798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45268 (local address 127.16.98.189:45139)
0807 22:50:12.416876 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:12.416878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:12.416961 (+ 83us) server_negotiation.cc:407] Connection header received
0807 22:50:12.417121 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:12.417122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:12.417135 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:12.417213 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:12.417616 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45268: BlockingRecv error: recv got EOF from 127.0.0.1:45268 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 22:50:12.713629 7417 catalog_manager.cc:1259] Loaded cluster ID: 72650c22753d442a8d5043cc73cd0c83
I20260807 22:50:12.713717 7417 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563: loading cluster ID for follower catalog manager: success
I20260807 22:50:12.714572 7417 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563: acquiring CA information for follower catalog manager: success
I20260807 22:50:12.714915 7417 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d26aee0781474840a561b07957574563: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 22:50:12.758553 7432 catalog_manager.cc:1259] Loaded cluster ID: 72650c22753d442a8d5043cc73cd0c83
I20260807 22:50:12.758637 7432 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631: loading cluster ID for follower catalog manager: success
I20260807 22:50:12.759570 7432 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631: acquiring CA information for follower catalog manager: success
I20260807 22:50:12.759902 7432 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f8f8f90c6704405bb67d63957d094631: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 22:50:12.934149 7533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:12.931175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.16.98.189:45139)
0807 22:50:12.931415 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:12.931417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:12.931434 (+ 17us) server_negotiation.cc:407] Connection header received
0807 22:50:12.931467 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:12.931467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:12.931519 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:12.931599 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:50:12.931803 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:12.932329 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:12.933213 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:12.933454 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:12.933519 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:12.933724 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:50:12.933849 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:50:12.933942 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:12.934039 (+ 97us) server_negotiation.cc:299] Negotiation successful
0807 22:50:12.934063 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":112,"threads_started":1}
I20260807 22:50:12.938977 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:12.936296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60006 (local address 127.16.98.188:40899)
0807 22:50:12.936436 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:12.936438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:12.936447 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:50:12.936504 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:12.936505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:12.936546 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:12.936681 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:50:12.936804 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:12.937322 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:12.938100 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:12.938330 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:12.938396 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:12.938545 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:50:12.938645 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:50:12.938704 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:12.938790 (+ 86us) server_negotiation.cc:299] Negotiation successful
0807 22:50:12.938810 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":85,"threads_started":1}
I20260807 22:50:12.944271 7533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:12.940962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.16.98.189:45139)
0807 22:50:12.941009 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:12.941011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:12.941193 (+ 182us) server_negotiation.cc:407] Connection header received
0807 22:50:12.941313 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:12.941314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:12.941328 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:12.941362 (+ 34us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:50:12.941565 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:12.942400 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:12.943423 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:12.943730 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:12.943798 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:12.943949 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:50:12.943983 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:50:12.944038 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:12.944140 (+ 102us) server_negotiation.cc:299] Negotiation successful
0807 22:50:12.944170 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 22:50:12.944928 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:12.940966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54234 (local address 127.16.98.190:40441)
0807 22:50:12.941071 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:12.941073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:12.941190 (+ 117us) server_negotiation.cc:407] Connection header received
0807 22:50:12.941338 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:12.941339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:12.941401 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:12.941505 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:50:12.941711 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:12.942467 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:12.943718 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:12.944060 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:12.944126 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:12.944314 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:50:12.944484 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:50:12.944676 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:12.944732 (+ 56us) server_negotiation.cc:299] Negotiation successful
0807 22:50:12.944760 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":47,"threads_started":1}
I20260807 22:50:12.945376 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:12.941213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60018 (local address 127.16.98.188:40899)
0807 22:50:12.941266 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:12.941268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:12.941462 (+ 194us) server_negotiation.cc:407] Connection header received
0807 22:50:12.941492 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:12.941494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:12.941506 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:12.941566 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:50:12.942637 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:12.943404 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:12.944504 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:12.944825 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:12.944868 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:12.945111 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:50:12.945145 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:50:12.945236 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:12.945294 (+ 58us) server_negotiation.cc:299] Negotiation successful
0807 22:50:12.945317 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260807 22:50:12.946682 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 7170
I20260807 22:50:12.953826 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 7249
I20260807 22:50:12.960379 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 7329
2026-08-07T22:50:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3086 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 22:50:13 dist-test-slave-rgb1 krb5kdc[7550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 22:50:13 dist-test-slave-rgb1 krb5kdc[7550](info): set up 2 sockets
Aug 07 22:50:13 dist-test-slave-rgb1 krb5kdc[7550](info): commencing operation
krb5kdc: starting...
W20260807 22:50:14.056749 16778 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/test-user.keytab.
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T22:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.98.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
Entry for principal kudu/127.16.98.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab.
I20260807 22:50:14.201798 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:42315
--webserver_interface=127.16.98.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:42315,127.16.98.189:40637,127.16.98.188:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 22:50:14.289508 7563 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 22:50:14.289844 7563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:50:14.289883 7563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:50:14.289937 7563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:50:14.292196 7563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:50:14.292265 7563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:50:14.292291 7563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:50:14.292323 7563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:50:14.292361 7563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:50:14.295078 7563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:42315,127.16.98.189:40637,127.16.98.188:34691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.190.keytab
--principal=kudu/127.16.98.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.190:42315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.98.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:50:14.295380 7563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:50:14.295645 7563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:50:14.301537 7569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:14.301554 7571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:14.301900 7568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:14.301899 7563 server_base.cc:1034] running on GCE node
I20260807 22:50:14.302300 7563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:50:14.302632 7563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:50:14.303802 7563 hybrid_clock.cc:648] HybridClock initialized: now 1786143014303786 us; error 35 us; skew 500 ppm
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:50:14.306965 7563 init.cc:377] Logged in from keytab as kudu/127.16.98.190@KRBTEST.COM (short username kudu)
I20260807 22:50:14.307539 7563 webserver.cc:460] Webserver started at http://127.16.98.190:45715/ using document root <none> and password file <none>
I20260807 22:50:14.307821 7563 fs_manager.cc:359] Metadata directory not provided
I20260807 22:50:14.307890 7563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:50:14.308053 7563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:50:14.309671 7563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/data/instance:
uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4"
format_stamp: "Formatted at 2026-08-07 22:50:14 on dist-test-slave-rgb1"
server_key: "000e043fe2914d57d00240a09387a8d4"
server_key_iv: "7629f8e7c232dcd7f7a1d3f9d94d56ef"
server_key_version: "encryptionkey@0"
I20260807 22:50:14.310181 7563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/wal/instance:
uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4"
format_stamp: "Formatted at 2026-08-07 22:50:14 on dist-test-slave-rgb1"
server_key: "000e043fe2914d57d00240a09387a8d4"
server_key_iv: "7629f8e7c232dcd7f7a1d3f9d94d56ef"
server_key_version: "encryptionkey@0"
I20260807 22:50:14.312183 7563 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260807 22:50:14.313288 7578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:50:14.313591 7563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 22:50:14.313726 7563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/wal
uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4"
format_stamp: "Formatted at 2026-08-07 22:50:14 on dist-test-slave-rgb1"
server_key: "000e043fe2914d57d00240a09387a8d4"
server_key_iv: "7629f8e7c232dcd7f7a1d3f9d94d56ef"
server_key_version: "encryptionkey@0"
I20260807 22:50:14.313832 7563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:50:14.348125 7563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:50:14.350639 7563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:50:14.350824 7563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:50:14.355242 7563 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.190:42315
I20260807 22:50:14.355281 7634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.190:42315 every 8 connection(s)
I20260807 22:50:14.355681 7563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/data/info.pb
I20260807 22:50:14.356917 7635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:50:14.358534 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 7563
I20260807 22:50:14.358654 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-0/wal/instance
I20260807 22:50:14.358892 16778 external_mini_cluster.cc:1428] Setting key 2a242e15c8bb677dfa286a8ab9ad82fe
I20260807 22:50:14.358969 7635 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.98.189@KRBTEST.COM; defaulting to no policy
I20260807 22:50:14.370800 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.359890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:42315 (local address 127.0.0.1:48220)
0807 22:50:14.360113 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.360159 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:14.360735 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:14.361028 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:14.361031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:14.361349 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:14.361508 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.361512 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.362547 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.362548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.363249 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.363251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.363795 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.363797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.363860 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.364524 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:14.364533 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:14.366441 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:14.368882 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.368890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.368893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.369174 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.369472 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.369473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.369474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.369577 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.370019 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:14.370028 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:14.370286 (+ 258us) client_negotiation.cc:769] Sending connection context
0807 22:50:14.370410 (+ 124us) client_negotiation.cc:240] Negotiation successful
0807 22:50:14.370586 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":130,"threads_started":1}
I20260807 22:50:14.370805 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.360144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48220 (local address 127.16.98.190:42315)
0807 22:50:14.360315 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.360321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.360364 (+ 43us) server_negotiation.cc:407] Connection header received
0807 22:50:14.360903 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.360905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.360923 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.361021 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.361613 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.362435 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.363350 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.363680 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.363800 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.366542 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:14.366552 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:14.366552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:14.366562 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:14.368779 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.369274 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.369278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.369278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.369367 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.369673 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.369674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.369674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.369920 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:14.370021 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:14.370462 (+ 441us) server_negotiation.cc:299] Negotiation successful
0807 22:50:14.370663 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260807 22:50:14.371974 7635 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:50:14.372483 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.372320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40637 (local address 127.0.0.1:45164)
0807 22:50:14.372395 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.372446 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260807 22:50:14.372812 7582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40637: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.189@KRBTEST.COM" created.
I20260807 22:50:14.373395 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.373257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40637 (local address 127.0.0.1:45170)
0807 22:50:14.373318 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.373358 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260807 22:50:14.373533 7635 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.189:40637: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40637: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.189.keytab.
Entry for principal kudu/127.16.98.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.189.keytab.
I20260807 22:50:14.392485 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.189:40637
--webserver_interface=127.16.98.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.189.keytab
--principal=kudu/127.16.98.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:42315,127.16.98.189:40637,127.16.98.188:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:50:14.422822 7635 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } attempt: 1
I20260807 22:50:14.423434 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.423285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40637 (local address 127.0.0.1:45180)
0807 22:50:14.423360 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.423403 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260807 22:50:14.423918 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.423827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40637 (local address 127.0.0.1:45190)
0807 22:50:14.423872 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.423900 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260807 22:50:14.424144 7635 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.189:40637: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40637: connect: Connection refused (error 111)
W20260807 22:50:14.479959 7642 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 22:50:14.480183 7642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:50:14.480201 7642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:50:14.480221 7642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:50:14.481462 7642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:50:14.481508 7642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:50:14.481523 7642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:50:14.481534 7642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:50:14.481547 7642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:50:14.483191 7642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:42315,127.16.98.189:40637,127.16.98.188:34691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.189.keytab
--principal=kudu/127.16.98.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.189:40637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.98.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:50:14.483479 7642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:50:14.483731 7642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:50:14.489080 7647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:14.489176 7648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:14.489377 7650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:14.489902 7642 server_base.cc:1034] running on GCE node
I20260807 22:50:14.490199 7642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:50:14.490561 7642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:50:14.491748 7642 hybrid_clock.cc:648] HybridClock initialized: now 1786143014491722 us; error 37 us; skew 500 ppm
I20260807 22:50:14.492314 7635 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } attempt: 2
I20260807 22:50:14.492795 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.492657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40637 (local address 127.0.0.1:45200)
0807 22:50:14.492720 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.492753 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260807 22:50:14.493196 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.493092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40637 (local address 127.0.0.1:45212)
0807 22:50:14.493141 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.493166 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260807 22:50:14.493348 7635 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.189:40637: Network error: Client connection negotiation failed: client connection to 127.16.98.189:40637: connect: Connection refused (error 111)
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:50:14.494967 7642 init.cc:377] Logged in from keytab as kudu/127.16.98.189@KRBTEST.COM (short username kudu)
I20260807 22:50:14.495517 7642 webserver.cc:460] Webserver started at http://127.16.98.189:42771/ using document root <none> and password file <none>
I20260807 22:50:14.495880 7642 fs_manager.cc:359] Metadata directory not provided
I20260807 22:50:14.495959 7642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:50:14.496121 7642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:50:14.497313 7642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/data/instance:
uuid: "383541d6181349eeb2607e106a1f3829"
format_stamp: "Formatted at 2026-08-07 22:50:14 on dist-test-slave-rgb1"
server_key: "7d68cd4b498b45f207d509f3d4f420a1"
server_key_iv: "cb9a98cf77dc18b3d0b447c5bdf367cf"
server_key_version: "encryptionkey@0"
I20260807 22:50:14.497699 7642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/wal/instance:
uuid: "383541d6181349eeb2607e106a1f3829"
format_stamp: "Formatted at 2026-08-07 22:50:14 on dist-test-slave-rgb1"
server_key: "7d68cd4b498b45f207d509f3d4f420a1"
server_key_iv: "cb9a98cf77dc18b3d0b447c5bdf367cf"
server_key_version: "encryptionkey@0"
I20260807 22:50:14.499382 7642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260807 22:50:14.500353 7657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:50:14.500648 7642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:50:14.500757 7642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/wal
uuid: "383541d6181349eeb2607e106a1f3829"
format_stamp: "Formatted at 2026-08-07 22:50:14 on dist-test-slave-rgb1"
server_key: "7d68cd4b498b45f207d509f3d4f420a1"
server_key_iv: "cb9a98cf77dc18b3d0b447c5bdf367cf"
server_key_version: "encryptionkey@0"
I20260807 22:50:14.500854 7642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:50:14.513695 7642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:50:14.516141 7642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:50:14.516316 7642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:50:14.520756 7642 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.189:40637
I20260807 22:50:14.520812 7711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.189:40637 every 8 connection(s)
I20260807 22:50:14.521214 7642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/data/info.pb
I20260807 22:50:14.522455 7712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:50:14.524179 7712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:50:14.529134 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 7642
I20260807 22:50:14.529269 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-1/wal/instance
I20260807 22:50:14.529520 16778 external_mini_cluster.cc:1428] Setting key 5742e76163a16fd82dff23d9fede0a8b
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.189@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:50:14.534574 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.525130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.16.98.190:42315)
0807 22:50:14.525233 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.525236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.525340 (+ 104us) server_negotiation.cc:407] Connection header received
0807 22:50:14.525840 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.525841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.525856 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.525913 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.526627 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.527165 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.528139 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.528392 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.528467 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.531167 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:14.531169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:14.531169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:14.531178 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:14.532916 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.533441 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.533443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.533443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.533515 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.533815 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.533816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.533816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.533959 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:14.534039 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:14.534370 (+ 331us) server_negotiation.cc:299] Negotiation successful
0807 22:50:14.534469 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 22:50:14.534802 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.525009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:42315 (local address 127.0.0.1:48228)
0807 22:50:14.525211 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.525250 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:14.525743 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:14.526064 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:14.526068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:14.526323 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:14.526528 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.526532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.527260 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.527261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.528034 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.528036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.528506 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.528507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.528676 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.529288 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:14.529299 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:14.531034 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:14.533044 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.533052 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.533054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.533334 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.533594 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.533595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.533596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.533725 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.534044 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:14.534052 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:14.534215 (+ 163us) client_negotiation.cc:769] Sending connection context
0807 22:50:14.534340 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 22:50:14.534526 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
I20260807 22:50:14.535472 7712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.98.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.98.188@KRBTEST.COM" created.
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.189@KRBTEST.COM for kudu/127.16.98.189@KRBTEST.COM
I20260807 22:50:14.545496 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.535836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40637 (local address 127.0.0.1:45228)
0807 22:50:14.535914 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.535926 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:14.535971 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:14.536238 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:14.536239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:14.536398 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:14.536454 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.536456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.537472 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.537473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.538105 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.538107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.538667 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.538668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.538740 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.539233 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:14.539243 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:14.542172 (+ 2929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:14.544263 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.544264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.544265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.544440 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.544761 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.544762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.544763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.544827 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.545145 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:14.545146 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:14.545187 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 22:50:14.545258 (+ 71us) client_negotiation.cc:240] Negotiation successful
0807 22:50:14.545377 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260807 22:50:14.545531 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.535885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45228 (local address 127.16.98.189:40637)
0807 22:50:14.536072 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.536077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.536103 (+ 26us) server_negotiation.cc:407] Connection header received
0807 22:50:14.536122 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.536124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.536135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.536220 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.536548 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.537365 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.538203 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.538567 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.538668 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.542302 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:14.542315 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:14.542315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:14.542331 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:14.544134 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.544544 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.544547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.544547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.544662 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.544894 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.544895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.544895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.545072 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:14.545171 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:14.545300 (+ 129us) server_negotiation.cc:299] Negotiation successful
0807 22:50:14.545434 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":162,"thread_start_us":98,"threads_started":1}
I20260807 22:50:14.546380 7712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:50:14.546824 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.546671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:34691 (local address 127.0.0.1:54114)
0807 22:50:14.546744 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.546796 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 22:50:14.547017 7658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
I20260807 22:50:14.547634 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.547518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:34691 (local address 127.0.0.1:54120)
0807 22:50:14.547573 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.547600 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260807 22:50:14.547742 7712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:34691: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.98.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.188.keytab.
Entry for principal kudu/127.16.98.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.188.keytab.
I20260807 22:50:14.560775 16778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
/tmp/dist-test-taskG4wJ7J/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.188:34691
--webserver_interface=127.16.98.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.188.keytab
--principal=kudu/127.16.98.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.98.148:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.98.190:42315,127.16.98.189:40637,127.16.98.188:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 22:50:14.584509 7635 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } attempt: 3
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for kudu/127.16.98.189@KRBTEST.COM
I20260807 22:50:14.593315 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.585159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45240 (local address 127.16.98.189:40637)
0807 22:50:14.585231 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.585233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.585245 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:50:14.585304 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.585305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.585318 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.585392 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.585773 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.586538 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.587474 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.587723 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.587806 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.590224 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:14.590225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:14.590226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:14.590237 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:14.591821 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.592364 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.592365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.592366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.592435 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.592713 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.592714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.592715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.592851 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:14.592949 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:14.593114 (+ 165us) server_negotiation.cc:299] Negotiation successful
0807 22:50:14.593206 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 22:50:14.593315 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.585086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40637 (local address 127.0.0.1:45240)
0807 22:50:14.585176 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.585189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:14.585240 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:14.585398 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:14.585399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:14.585601 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:14.585677 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.585680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.586641 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.586642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.587385 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.587388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.587803 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.587804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.587882 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.588432 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:14.588438 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:14.590082 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:14.592043 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.592045 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.592046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.592260 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.592516 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.592517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.592517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.592630 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.592931 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:14.592932 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:14.592987 (+ 55us) client_negotiation.cc:769] Sending connection context
0807 22:50:14.593085 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 22:50:14.593202 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260807 22:50:14.594084 7635 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:50:14.594514 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.594362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:34691 (local address 127.0.0.1:54122)
0807 22:50:14.594416 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.594481 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260807 22:50:14.594897 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.594797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:34691 (local address 127.0.0.1:54132)
0807 22:50:14.594844 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.594873 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260807 22:50:14.595103 7635 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:34691: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
I20260807 22:50:14.597031 7712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } attempt: 1
I20260807 22:50:14.597463 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.597337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:34691 (local address 127.0.0.1:54146)
0807 22:50:14.597397 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.597429 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260807 22:50:14.597823 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.597724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:34691 (local address 127.0.0.1:54158)
0807 22:50:14.597773 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.597800 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260807 22:50:14.597929 7712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:34691: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
I20260807 22:50:14.626289 7635 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } attempt: 1
I20260807 22:50:14.626832 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.626659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:34691 (local address 127.0.0.1:54164)
0807 22:50:14.626736 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.626789 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260807 22:50:14.627260 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.627159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:34691 (local address 127.0.0.1:54180)
0807 22:50:14.627208 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.627238 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260807 22:50:14.627378 7635 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:34691: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
W20260807 22:50:14.650012 7719 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 22:50:14.650271 7719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:50:14.650315 7719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:50:14.650358 7719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:50:14.651495 7719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:50:14.651564 7719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 22:50:14.651592 7719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:50:14.651623 7719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:50:14.651651 7719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:50:14.653270 7719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.98.148:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.98.190:42315,127.16.98.189:40637,127.16.98.188:34691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/krb5kdc/kudu_127.16.98.188.keytab
--principal=kudu/127.16.98.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.98.188:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.98.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 07 Aug 2026 21:52:04 UTC on ccycloud.quasar-kxqltd.root.comops.site
build id 4573644
I20260807 22:50:14.653559 7719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:50:14.653796 7719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:50:14.658926 7724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:14.659045 7727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:50:14.658931 7725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:50:14.659345 7719 server_base.cc:1034] running on GCE node
I20260807 22:50:14.659617 7719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:50:14.659894 7719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:50:14.661077 7719 hybrid_clock.cc:648] HybridClock initialized: now 1786143014661094 us; error 61 us; skew 500 ppm
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 22:50:14.663919 7719 init.cc:377] Logged in from keytab as kudu/127.16.98.188@KRBTEST.COM (short username kudu)
I20260807 22:50:14.664426 7719 webserver.cc:460] Webserver started at http://127.16.98.188:35651/ using document root <none> and password file <none>
I20260807 22:50:14.664714 7719 fs_manager.cc:359] Metadata directory not provided
I20260807 22:50:14.664788 7719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:50:14.664949 7719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:50:14.666091 7712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } attempt: 2
I20260807 22:50:14.666076 7719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/data/instance:
uuid: "4d15160c57ee4abe821b57e65c00c0f9"
format_stamp: "Formatted at 2026-08-07 22:50:14 on dist-test-slave-rgb1"
server_key: "e63aafa52bd5e79e17274648726e40e9"
server_key_iv: "ef110fe8acca468365902136082e4b01"
server_key_version: "encryptionkey@0"
I20260807 22:50:14.666503 7719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/wal/instance:
uuid: "4d15160c57ee4abe821b57e65c00c0f9"
format_stamp: "Formatted at 2026-08-07 22:50:14 on dist-test-slave-rgb1"
server_key: "e63aafa52bd5e79e17274648726e40e9"
server_key_iv: "ef110fe8acca468365902136082e4b01"
server_key_version: "encryptionkey@0"
I20260807 22:50:14.666564 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.666415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:34691 (local address 127.0.0.1:54192)
0807 22:50:14.666483 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.666525 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260807 22:50:14.666965 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.666845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:34691 (local address 127.0.0.1:54204)
0807 22:50:14.666897 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.666940 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260807 22:50:14.667104 7712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.98.188:34691: Network error: Client connection negotiation failed: client connection to 127.16.98.188:34691: connect: Connection refused (error 111)
I20260807 22:50:14.668274 7719 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260807 22:50:14.669272 7734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:50:14.669565 7719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 22:50:14.669691 7719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/data,/tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/wal
uuid: "4d15160c57ee4abe821b57e65c00c0f9"
format_stamp: "Formatted at 2026-08-07 22:50:14 on dist-test-slave-rgb1"
server_key: "e63aafa52bd5e79e17274648726e40e9"
server_key_iv: "ef110fe8acca468365902136082e4b01"
server_key_version: "encryptionkey@0"
I20260807 22:50:14.669801 7719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:50:14.688783 7719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:50:14.691185 7719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:50:14.691381 7719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:50:14.696833 7719 rpc_server.cc:307] RPC server started. Bound to: 127.16.98.188:34691
I20260807 22:50:14.696858 7787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.98.188:34691 every 8 connection(s)
I20260807 22:50:14.697327 7719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/data/info.pb
I20260807 22:50:14.698580 7788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:50:14.700388 7788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:50:14.702531 7635 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } attempt: 2
I20260807 22:50:14.707319 16778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu as pid 7719
I20260807 22:50:14.707448 16778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG4wJ7J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786142716408518-16778-0/minicluster-data/master-2/wal/instance
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.188@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:50:14.707666 16778 external_mini_cluster.cc:1428] Setting key cc10858f01ffcdb43d0d6c6258446ac3
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.190@KRBTEST.COM for kudu/127.16.98.188@KRBTEST.COM
I20260807 22:50:14.712203 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.702904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:34691 (local address 127.0.0.1:54216)
0807 22:50:14.702968 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.702978 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:14.703019 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:14.703332 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:14.703333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:14.703486 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:14.703522 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.703524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.705019 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.705020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.705712 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.705714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.706360 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.706361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.706406 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.706809 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:14.706814 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:14.708901 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:14.711072 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.711073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.711074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.711287 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.711494 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.711495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.711495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.711556 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.711908 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:14.711909 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:14.711965 (+ 56us) client_negotiation.cc:769] Sending connection context
0807 22:50:14.712046 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 22:50:14.712130 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260807 22:50:14.712352 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.701258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.190:42315 (local address 127.0.0.1:48238)
0807 22:50:14.701490 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.701531 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:14.702066 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:14.702385 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:14.702389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:14.702839 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:14.703155 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.703162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.703866 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.703867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.705069 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.705071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.705572 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.705573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.705704 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.706379 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:14.706394 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:14.708147 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:14.710466 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.710472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.710474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.710823 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.711091 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.711092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.711093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.711241 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.711614 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:14.711622 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:14.711790 (+ 168us) client_negotiation.cc:769] Sending connection context
0807 22:50:14.711891 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 22:50:14.712077 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":127,"threads_started":1}
I20260807 22:50:14.712700 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.701286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48238 (local address 127.16.98.190:42315)
0807 22:50:14.701356 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.701358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.701631 (+ 273us) server_negotiation.cc:407] Connection header received
0807 22:50:14.702196 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.702198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.702211 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.702268 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.703266 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.703749 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.705138 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.705433 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.705513 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.708237 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:14.708238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:14.708239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:14.708247 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:14.710366 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.710917 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.710918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.710919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.710998 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.711317 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.711318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.711318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.711537 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:14.711614 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:14.712479 (+ 865us) server_negotiation.cc:299] Negotiation successful
0807 22:50:14.712617 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 22:50:14.712751 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.702966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.16.98.188:34691)
0807 22:50:14.703157 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.703160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.703178 (+ 18us) server_negotiation.cc:407] Connection header received
0807 22:50:14.703196 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.703198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.703217 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.703301 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.703609 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.704916 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.705797 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.706253 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.706336 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.709013 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:14.709021 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:14.709021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:14.709027 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:14.710979 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.711368 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.711371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.711371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.711421 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.711654 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.711654 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.711655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.711826 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:14.711908 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:14.712098 (+ 190us) server_negotiation.cc:299] Negotiation successful
0807 22:50:14.712164 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":162,"thread_start_us":120,"threads_started":1}
I20260807 22:50:14.713181 7788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.188@KRBTEST.COM
I20260807 22:50:14.715025 7635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4: Bootstrap starting.
I20260807 22:50:14.716150 7635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4: Neither blocks nor log segments found. Creating new log.
I20260807 22:50:14.716527 7635 log.cc:824] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4: Log is configured to *not* fsync() on all Append() calls
I20260807 22:50:14.717588 7635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4: No bootstrap required, opened a new log
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.188@KRBTEST.COM for kudu/127.16.98.189@KRBTEST.COM
I20260807 22:50:14.719556 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.708352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.16.98.188:34691)
0807 22:50:14.708527 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.708529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.708537 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:50:14.708596 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.708597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.708609 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.708680 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.709580 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.710479 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.711550 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.711829 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.711915 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.715388 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:14.715389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:14.715390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:14.715398 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:14.718228 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.718642 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.718643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.718643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.718698 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.718947 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.718948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.718948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.719096 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:14.719306 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:14.719348 (+ 42us) server_negotiation.cc:299] Negotiation successful
0807 22:50:14.719433 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":116,"threads_started":1}
I20260807 22:50:14.720054 7635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } }
I20260807 22:50:14.720216 7635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3250391ef8c14dbe9fd0b2e2a0a5a8e4, State: Initialized, Role: FOLLOWER
I20260807 22:50:14.720304 7635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } }
I20260807 22:50:14.720641 7796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } } }
I20260807 22:50:14.720767 7796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:50:14.720703 7635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 22:50:14.722915 7808 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:50:14.724114 7808 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:50:14.724390 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.713509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.189:40637 (local address 127.0.0.1:45242)
0807 22:50:14.713559 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.713570 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:14.713610 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:14.713736 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:14.713737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:14.713893 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:14.713924 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.713925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.714634 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.714635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.715391 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.715393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.715962 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.715963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.716069 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.716487 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:14.716493 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:14.721013 (+ 4520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:14.722998 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.722999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.722999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.723167 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.723400 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.723401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.723402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.723471 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.723783 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:14.723784 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:14.723831 (+ 47us) client_negotiation.cc:769] Sending connection context
0807 22:50:14.723895 (+ 64us) client_negotiation.cc:240] Negotiation successful
0807 22:50:14.723996 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260807 22:50:14.724725 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.713572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.16.98.189:40637)
0807 22:50:14.713620 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.713622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.713634 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:50:14.713666 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.713667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.713678 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.713735 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.714024 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.714537 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.715478 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.715837 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.715903 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.721133 (+ 5230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:14.721135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:14.721135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:14.721143 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:14.722863 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.723251 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.723253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.723253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.723319 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.723543 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.723544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.723544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.723687 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:14.723752 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:14.724522 (+ 770us) server_negotiation.cc:299] Negotiation successful
0807 22:50:14.724634 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.190@KRBTEST.COM
I20260807 22:50:14.724709 7563 master_runner.cc:186] Error getting master registration for 127.16.98.189:40637: OK, instance_id { permanent_uuid: "383541d6181349eeb2607e106a1f3829" instance_seqno: 1786143014519545 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 22:50:14.727638 7563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 22:50:14.728026 7788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } has no permanent_uuid. Determining permanent_uuid...
I20260807 22:50:14.730095 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.720254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.16.98.190:42315)
0807 22:50:14.720300 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.720302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.720311 (+ 9us) server_negotiation.cc:407] Connection header received
0807 22:50:14.720376 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.720377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.720388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.720448 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.721125 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.722307 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.723374 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.723709 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.723777 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.726777 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:14.726779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:14.726779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:14.726787 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:14.728765 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.729219 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.729221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.729221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.729286 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.729532 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.729532 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.729533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.729677 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:14.729837 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:14.729904 (+ 67us) server_negotiation.cc:299] Negotiation successful
0807 22:50:14.729986 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.188@KRBTEST.COM for kudu/127.16.98.188@KRBTEST.COM
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.98.189@KRBTEST.COM
I20260807 22:50:14.736397 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.728346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:34691 (local address 127.0.0.1:54242)
0807 22:50:14.728410 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.728421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:14.728469 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:14.728654 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:14.728654 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:14.728793 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:14.728828 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.728830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.729589 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.729590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.730449 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.730452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.731000 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.731001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.731090 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.731550 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:14.731556 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:14.732902 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:14.735036 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.735037 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.735037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.735255 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.735533 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.735534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.735535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.735609 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.736000 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:14.736001 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:14.736052 (+ 51us) client_negotiation.cc:769] Sending connection context
0807 22:50:14.736136 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 22:50:14.736254 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 22:50:14.736491 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.728377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54242 (local address 127.16.98.188:34691)
0807 22:50:14.728414 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.728416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.728449 (+ 33us) server_negotiation.cc:407] Connection header received
0807 22:50:14.728513 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.728514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.728525 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.728639 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.728914 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.729518 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.730552 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.730917 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.731001 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.733004 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:14.733006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:14.733006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:14.733014 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:14.734935 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.735348 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.735350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.735350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.735445 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.735687 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.735688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.735688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.735905 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:14.735993 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:14.736169 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 22:50:14.736328 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":16}
I20260807 22:50:14.738885 7788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9: Bootstrap starting.
I20260807 22:50:14.739120 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.730670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45256 (local address 127.16.98.189:40637)
0807 22:50:14.730710 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.730712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.730752 (+ 40us) server_negotiation.cc:407] Connection header received
0807 22:50:14.730826 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.730826 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.730837 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.730881 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.731521 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.732300 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.733258 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.733578 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.733676 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.736223 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:14.736224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:14.736225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:14.736233 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:14.737909 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.738265 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.738266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.738266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.738327 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.738528 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.738529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.738529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.738647 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:14.738817 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:14.738855 (+ 38us) server_negotiation.cc:299] Negotiation successful
0807 22:50:14.738932 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260807 22:50:14.739703 7788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9: Neither blocks nor log segments found. Creating new log.
I20260807 22:50:14.740015 7788 log.cc:824] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9: Log is configured to *not* fsync() on all Append() calls
I20260807 22:50:14.740459 16778 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 22:50:14.741048 7788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9: No bootstrap required, opened a new log
I20260807 22:50:14.741343 7607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48242
I20260807 22:50:14.741505 7607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 22:50:14.741575 7607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:50:14.742309 7607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } }
I20260807 22:50:14.742604 7607 leader_election.cc:288] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [CANDIDATE]: Term 1 election: Requested vote from peers 383541d6181349eeb2607e106a1f3829 (127.16.98.189:40637), 4d15160c57ee4abe821b57e65c00c0f9 (127.16.98.188:34691)
I20260807 22:50:14.743304 7788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } }
I20260807 22:50:14.743470 7788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d15160c57ee4abe821b57e65c00c0f9, State: Initialized, Role: FOLLOWER
I20260807 22:50:14.743556 7788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } }
I20260807 22:50:14.743876 7788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:50:14.743968 7761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4d15160c57ee4abe821b57e65c00c0f9"
I20260807 22:50:14.744129 7761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:50:14.745004 7761 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3250391ef8c14dbe9fd0b2e2a0a5a8e4 in term 1.
I20260807 22:50:14.745256 7579 leader_election.cc:302] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3250391ef8c14dbe9fd0b2e2a0a5a8e4, 4d15160c57ee4abe821b57e65c00c0f9; no voters:
I20260807 22:50:14.745817 7796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:50:14.746011 7796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [term 1 LEADER]: Becoming Leader. State: Replica: 3250391ef8c14dbe9fd0b2e2a0a5a8e4, State: Running, Role: LEADER
I20260807 22:50:14.746119 7796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } }
I20260807 22:50:14.746598 7796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3250391ef8c14dbe9fd0b2e2a0a5a8e4. Latest consensus state: current_term: 1 leader_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } } }
I20260807 22:50:14.746687 7796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [sys.catalog]: This master's current role is: LEADER
W20260807 22:50:14.748167 7829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:50:14.748214 7829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:50:14.748049 7719 master_runner.cc:186] Error getting master registration for 127.16.98.189:40637: OK, instance_id { permanent_uuid: "383541d6181349eeb2607e106a1f3829" instance_seqno: 1786143014519545 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 22:50:14.750344 7719 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 22:50:14.752729 7833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:50:14.753021 7833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:50:14.755548 7761 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9 [term 1 FOLLOWER]: Refusing update from remote peer 3250391ef8c14dbe9fd0b2e2a0a5a8e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:50:14.755879 7796 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:50:14.758232 7712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } attempt: 3
I20260807 22:50:14.759600 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.743910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.16.98.190:42315)
0807 22:50:14.743982 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.743985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.747021 (+ 3036us) server_negotiation.cc:407] Connection header received
0807 22:50:14.747301 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.747302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.747313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.747375 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.748643 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.749359 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.750414 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.750664 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.750751 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.752283 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:14.752285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:14.752285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:14.752292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:14.754033 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.754469 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.754471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.754471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.754533 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.754745 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.754746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.754746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.754890 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:14.754992 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:14.759403 (+ 4411us) server_negotiation.cc:299] Negotiation successful
0807 22:50:14.759499 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 22:50:14.761600 7815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } } }
I20260807 22:50:14.761750 7815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:50:14.761823 7815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3250391ef8c14dbe9fd0b2e2a0a5a8e4. Latest consensus state: current_term: 1 leader_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } } }
I20260807 22:50:14.761873 7815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9 [sys.catalog]: This master's current role is: FOLLOWER
Aug 07 22:50:14 dist-test-slave-rgb1 krb5kdc[7550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786143014, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.98.189@KRBTEST.COM for kudu/127.16.98.188@KRBTEST.COM
I20260807 22:50:14.768347 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.745613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.16.98.188:34691)
0807 22:50:14.745656 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.745658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.754218 (+ 8560us) server_negotiation.cc:407] Connection header received
0807 22:50:14.754345 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.754346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.754359 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.754394 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.755171 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.755880 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.761507 (+ 5627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.761830 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.761890 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.763432 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:14.763433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:14.763433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:14.763457 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:14.766132 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.766922 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.766923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.766924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.767004 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.767432 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.767433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.767433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.767608 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:14.767705 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:14.768143 (+ 438us) server_negotiation.cc:299] Negotiation successful
0807 22:50:14.768228 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 22:50:14.769781 7830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } } }
I20260807 22:50:14.769922 7830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:14.770118 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.758590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.16.98.188:34691)
0807 22:50:14.758691 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.758693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.758701 (+ 8us) server_negotiation.cc:407] Connection header received
0807 22:50:14.758720 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.758720 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.758729 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.758780 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.759110 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.760482 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.761878 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.762204 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.762266 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.764939 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:14.764940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:14.764940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:14.764947 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:14.767582 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.768934 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.768935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.768935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.769011 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.769338 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.769339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.769340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.769684 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:14.769751 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:14.769913 (+ 162us) server_negotiation.cc:299] Negotiation successful
0807 22:50:14.769998 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 22:50:14.770476 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.758534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.98.188:34691 (local address 127.0.0.1:54256)
0807 22:50:14.758585 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0807 22:50:14.758596 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 22:50:14.758634 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:50:14.758819 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:50:14.758820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:50:14.758995 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0807 22:50:14.759034 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.759035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.760612 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.760613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.761780 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:50:14.761783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.762319 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.762320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 22:50:14.762381 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.763034 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 22:50:14.763049 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 22:50:14.764846 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 22:50:14.768656 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.768657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.768657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.768852 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.769100 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 22:50:14.769101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 22:50:14.769101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0807 22:50:14.769265 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 22:50:14.769745 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 22:50:14.769746 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 22:50:14.769811 (+ 65us) client_negotiation.cc:769] Sending connection context
0807 22:50:14.769875 (+ 64us) client_negotiation.cc:240] Negotiation successful
0807 22:50:14.770121 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260807 22:50:14.770463 7815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } } }
I20260807 22:50:14.770555 7815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:50:14.770972 7835 mvcc.cc:205] Tried to move back new op lower bound from 7316041788434186240 to 7316041788400951296. Current Snapshot: MvccSnapshot[applied={T|T < 7316041788434186240}]
I20260807 22:50:14.771585 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:14.745916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45258 (local address 127.16.98.189:40637)
0807 22:50:14.745977 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.745979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.755629 (+ 9650us) server_negotiation.cc:407] Connection header received
0807 22:50:14.755650 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.755651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.755662 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.755705 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.756453 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.757218 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.758243 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:14.758559 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:14.758629 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:14.762997 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 22:50:14.762999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 22:50:14.762999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 22:50:14.763008 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 22:50:14.768526 (+ 5518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.770542 (+ 2016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.770544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.770544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.770608 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 22:50:14.770848 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 22:50:14.770849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 22:50:14.770849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0807 22:50:14.771121 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 22:50:14.771210 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:14.771390 (+ 180us) server_negotiation.cc:299] Negotiation successful
0807 22:50:14.771468 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 22:50:14.772387 7712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829: Bootstrap starting.
I20260807 22:50:14.773245 7712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829: Neither blocks nor log segments found. Creating new log.
I20260807 22:50:14.773594 7712 log.cc:824] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829: Log is configured to *not* fsync() on all Append() calls
I20260807 22:50:14.773909 7837 mvcc.cc:205] Tried to move back new op lower bound from 7316041788434186240 to 7316041788400951296. Current Snapshot: MvccSnapshot[applied={T|T < 7316041788434186240}]
I20260807 22:50:14.774504 7712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829: No bootstrap required, opened a new log
I20260807 22:50:14.774940 7833 catalog_manager.cc:1347] Generated new cluster ID: 195986e480304d33b45df811b7fb1577
I20260807 22:50:14.774996 7833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:50:14.776682 7712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } }
I20260807 22:50:14.776813 7712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383541d6181349eeb2607e106a1f3829, State: Initialized, Role: FOLLOWER
I20260807 22:50:14.776878 7712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } }
I20260807 22:50:14.777173 7712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:50:14.778563 7685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "383541d6181349eeb2607e106a1f3829"
I20260807 22:50:14.778676 7685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:50:14.779585 7685 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3250391ef8c14dbe9fd0b2e2a0a5a8e4 in term 1.
I20260807 22:50:14.780457 7684 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829 [term 1 FOLLOWER]: Refusing update from remote peer 3250391ef8c14dbe9fd0b2e2a0a5a8e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 22:50:14.780881 7796 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:50:14.784693 7842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } } }
I20260807 22:50:14.784821 7842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:50:14.784883 7842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3250391ef8c14dbe9fd0b2e2a0a5a8e4. Latest consensus state: current_term: 1 leader_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } } }
I20260807 22:50:14.784943 7842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 22:50:14.784981 7842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } } }
I20260807 22:50:14.785032 7842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 22:50:14.786149 7853 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 22:50:14.786218 7853 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 22:50:14.793114 7855 mvcc.cc:205] Tried to move back new op lower bound from 7316041788434186240 to 7316041788400951296. Current Snapshot: MvccSnapshot[applied={T|T < 7316041788434186240}]
I20260807 22:50:14.801314 7796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3250391ef8c14dbe9fd0b2e2a0a5a8e4" member_type: VOTER last_known_addr { host: "127.16.98.190" port: 42315 } } peers { permanent_uuid: "383541d6181349eeb2607e106a1f3829" member_type: VOTER last_known_addr { host: "127.16.98.189" port: 40637 } } peers { permanent_uuid: "4d15160c57ee4abe821b57e65c00c0f9" member_type: VOTER last_known_addr { host: "127.16.98.188" port: 34691 } } }
I20260807 22:50:14.801450 7796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4 [sys.catalog]: This master's current role is: LEADER
I20260807 22:50:14.804540 7833 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:50:14.805357 7833 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:50:14.821307 7833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3250391ef8c14dbe9fd0b2e2a0a5a8e4: Generated new TSK 0
I20260807 22:50:14.821578 7833 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 22:50:14.848929 7716 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:14.847717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45264 (local address 127.16.98.189:40637)
0807 22:50:14.847767 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.847769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.848141 (+ 372us) server_negotiation.cc:407] Connection header received
0807 22:50:14.848254 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.848255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.848268 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.848300 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.848855 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45264: BlockingRecv error: recv got EOF from 127.0.0.1:45264 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260807 22:50:14.850229 7638 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:14.849321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.16.98.190:42315)
0807 22:50:14.849362 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.849364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.849686 (+ 322us) server_negotiation.cc:407] Connection header received
0807 22:50:14.849783 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.849784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.849795 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.849824 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.850160 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48258: BlockingRecv error: recv got EOF from 127.0.0.1:48258 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260807 22:50:14.851022 7791 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:14.849555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54268 (local address 127.16.98.188:34691)
0807 22:50:14.849609 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.849611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.850500 (+ 889us) server_negotiation.cc:407] Connection header received
0807 22:50:14.850598 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.850599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.850610 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.850640 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.850960 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54268: BlockingRecv error: recv got EOF from 127.0.0.1:54268 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260807 22:50:14.853397 7716 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:14.852538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45280 (local address 127.16.98.189:40637)
0807 22:50:14.852609 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.852611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.852856 (+ 245us) server_negotiation.cc:407] Connection header received
0807 22:50:14.852971 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.852972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.852993 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.853023 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.853353 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45280: BlockingRecv error: recv got EOF from 127.0.0.1:45280 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260807 22:50:14.856417 7716 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:14.855622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45292 (local address 127.16.98.189:40637)
0807 22:50:14.855656 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.855658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.855908 (+ 250us) server_negotiation.cc:407] Connection header received
0807 22:50:14.856004 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.856005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.856016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.856060 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.856369 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45292: BlockingRecv error: recv got EOF from 127.0.0.1:45292 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260807 22:50:14.860484 7716 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:14.859651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45304 (local address 127.16.98.189:40637)
0807 22:50:14.859688 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.859690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.859943 (+ 253us) server_negotiation.cc:407] Connection header received
0807 22:50:14.860053 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.860054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.860067 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.860098 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.860433 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45304: BlockingRecv error: recv got EOF from 127.0.0.1:45304 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260807 22:50:14.866592 7716 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:14.865700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.16.98.189:40637)
0807 22:50:14.865743 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.865745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.866000 (+ 255us) server_negotiation.cc:407] Connection header received
0807 22:50:14.866126 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.866127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.866139 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.866169 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.866535 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45320: BlockingRecv error: recv got EOF from 127.0.0.1:45320 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260807 22:50:14.876917 7716 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:14.875893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.16.98.189:40637)
0807 22:50:14.875943 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.875946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.876263 (+ 317us) server_negotiation.cc:407] Connection header received
0807 22:50:14.876373 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.876374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.876387 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.876419 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.876857 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45322: BlockingRecv error: recv got EOF from 127.0.0.1:45322 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260807 22:50:14.895360 7716 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:14.894332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.16.98.189:40637)
0807 22:50:14.894380 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.894382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.894704 (+ 322us) server_negotiation.cc:407] Connection header received
0807 22:50:14.894818 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.894820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.894833 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.894866 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.895296 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45332: BlockingRecv error: recv got EOF from 127.0.0.1:45332 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260807 22:50:14.931530 7716 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:14.929784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.16.98.189:40637)
0807 22:50:14.929858 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.929861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.930654 (+ 793us) server_negotiation.cc:407] Connection header received
0807 22:50:14.930802 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.930804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.930823 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.930859 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.931457 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45344: BlockingRecv error: recv got EOF from 127.0.0.1:45344 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260807 22:50:14.998191 7716 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:14.997246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45358 (local address 127.16.98.189:40637)
0807 22:50:14.997351 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:14.997354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:14.997524 (+ 170us) server_negotiation.cc:407] Connection header received
0807 22:50:14.997576 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:14.997577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:14.997599 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:14.997677 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:14.998094 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45358: BlockingRecv error: recv got EOF from 127.0.0.1:45358 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260807 22:50:15.128363 7716 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:15.127598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.16.98.189:40637)
0807 22:50:15.127698 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:15.127700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:15.127730 (+ 30us) server_negotiation.cc:407] Connection header received
0807 22:50:15.127807 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:15.127808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:15.127821 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:15.127894 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:15.128267 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45360: BlockingRecv error: recv got EOF from 127.0.0.1:45360 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260807 22:50:15.386641 7716 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:50:15.385759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45366 (local address 127.16.98.189:40637)
0807 22:50:15.385855 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:15.385857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:15.385964 (+ 107us) server_negotiation.cc:407] Connection header received
0807 22:50:15.386045 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:15.386046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:15.386063 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:15.386159 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 22:50:15.386548 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45366: BlockingRecv error: recv got EOF from 127.0.0.1:45366 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 22:50:15.748731 7829 catalog_manager.cc:1259] Loaded cluster ID: 195986e480304d33b45df811b7fb1577
I20260807 22:50:15.748818 7829 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9: loading cluster ID for follower catalog manager: success
I20260807 22:50:15.749698 7829 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9: acquiring CA information for follower catalog manager: success
I20260807 22:50:15.750026 7829 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d15160c57ee4abe821b57e65c00c0f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 22:50:15.786723 7853 catalog_manager.cc:1259] Loaded cluster ID: 195986e480304d33b45df811b7fb1577
I20260807 22:50:15.786811 7853 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829: loading cluster ID for follower catalog manager: success
I20260807 22:50:15.787693 7853 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829: acquiring CA information for follower catalog manager: success
I20260807 22:50:15.788075 7853 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 383541d6181349eeb2607e106a1f3829: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 22:50:15.903569 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:15.900516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.16.98.189:40637)
0807 22:50:15.900696 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:15.900698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:15.900710 (+ 12us) server_negotiation.cc:407] Connection header received
0807 22:50:15.900836 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:15.900838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:15.900896 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:15.901003 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:50:15.901184 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:15.901719 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:15.902613 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:15.902835 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:15.902903 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:15.903169 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:50:15.903298 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:50:15.903427 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:15.903448 (+ 21us) server_negotiation.cc:299] Negotiation successful
0807 22:50:15.903480 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":114,"threads_started":1}
I20260807 22:50:15.909933 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:15.906344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54274 (local address 127.16.98.188:34691)
0807 22:50:15.906508 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:15.906510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:15.906523 (+ 13us) server_negotiation.cc:407] Connection header received
0807 22:50:15.906584 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:15.906585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:15.906671 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:15.907062 (+ 391us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:50:15.907243 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:15.907750 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:15.908717 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:15.909060 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:15.909190 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:15.909492 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:50:15.909645 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:50:15.909729 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:15.909826 (+ 97us) server_negotiation.cc:299] Negotiation successful
0807 22:50:15.909857 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260807 22:50:15.915014 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:15.912179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54282 (local address 127.16.98.188:34691)
0807 22:50:15.912221 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:15.912223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:15.912363 (+ 140us) server_negotiation.cc:407] Connection header received
0807 22:50:15.912389 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:15.912390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:15.912401 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:15.912464 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:50:15.912727 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:15.913429 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:15.914242 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:15.914495 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:15.914564 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:15.914706 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:50:15.914744 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:50:15.914794 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:15.914884 (+ 90us) server_negotiation.cc:299] Negotiation successful
0807 22:50:15.914909 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260807 22:50:15.915251 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:15.911982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45384 (local address 127.16.98.189:40637)
0807 22:50:15.912034 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:15.912037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:15.912218 (+ 181us) server_negotiation.cc:407] Connection header received
0807 22:50:15.912350 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:15.912351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:15.912365 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:15.912426 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:50:15.912546 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:15.913358 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:15.914446 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:15.914731 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:15.914791 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:15.914947 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:50:15.914988 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:50:15.915021 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:15.915150 (+ 129us) server_negotiation.cc:299] Negotiation successful
0807 22:50:15.915180 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260807 22:50:15.915642 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 22:50:15.911946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48264 (local address 127.16.98.190:42315)
0807 22:50:15.912065 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0807 22:50:15.912067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0807 22:50:15.912222 (+ 155us) server_negotiation.cc:407] Connection header received
0807 22:50:15.912279 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 22:50:15.912280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 22:50:15.912342 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 22:50:15.912429 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 22:50:15.912724 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:15.913217 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:15.914616 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 22:50:15.914923 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 22:50:15.914983 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 22:50:15.915240 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 22:50:15.915371 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 22:50:15.915430 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 22:50:15.915521 (+ 91us) server_negotiation.cc:299] Negotiation successful
0807 22:50:15.915553 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":49,"threads_started":1}
I20260807 22:50:15.917689 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 7563
I20260807 22:50:15.925081 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 7642
I20260807 22:50:15.931509 16778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG4wJ7J/build/release/bin/kudu with pid 7719
2026-08-07T22:50:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2970 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6057 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (299521 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST